We are all used to common content pagination, where you have tons of links pointing to other pages with more results. The main reason behind this is performance, after all, you’re not mad to render tons of records at once and slow down your application on each gigantic request. But then we come to a discussion about usability and how wouldn’t it be a better user experience to load more records asynchronously as the user scrolls down instead of requiring him to click a link and re-render the entire page, specially when it comes to things such as feeds, where people are already scrolling down a ton of content and wouldn’t like to hit any sort of ‘STOP’ sign on their scrolling.
After spending some time dealing with this on my side project, I came up with a nice and simple endless page scrolling solution with jQuery and Rails 3. I created an example application for a more practical view on our problem and hosted all its code at this Github repository for anyone to check out.
First off, the whole endless scrolling Javascript front-end is a courtesy of this plugin, although it lacks a huge improvement which I’ll talk about later, so case you want to use it, I recommend downloading my version instead.
On the back-end side of things, I decided it was better to to ditch pagination plugins such as will_paginate altogether and load records not based on an ‘offset’ parameter, but on a ‘last timestamp’ parameter, this way if the feed happens to update while you’re scrolling down you won’t get duplicated results on the bottom due to the database offset pushing older (and already rendered) records to the bottom, thus loading them again. Also, it’s important to somehow stop the endless scrolling when there are no more results to show, so the user doesn’t spam our server with useless requests. So we basically need to manage 3 things: appending new records to our already existent list through AJAX, update the ‘last timestamp’ parameter somewhere with each re-render and know when to stop asking for more.
Let’s get down to the code, starting with the JS:
$('ul').endlessScroll({ fireOnce: true, fireDelay: 500, ceaseFire: function(){ return $('#infinite-scroll').length ? false : true; }, callback: function(){ $.ajax({ url: '/posts', data: { last: $(this).attr('last') }, dataType: 'script' }); } });
Assuming we have an UL with a fixed height, an overflow:auto and a ‘last’ attribute, this code will make requests to the specified URL on each scroll event (you can configure the scrolling distance with the bottomPixels property) and stop making them when we don’t have an #infinite-scroll div. The original plugin would not check for this ceaseFire condition on EACH scrolling event, which is crucial here (or else, what’s the point in having it?), that’s why using my patched version is key. This is how our page should look like to be ready to receive the upcoming updates:
<% unless @posts.blank?%> <ul class='list' last="<%=@posts.to_a.last.created_at%>"> <%=render :partial => "post", :collection => @posts%> <div id="infinite-scroll"></div> </ul> <% end %>
Now we move to the back-end, where we need to respond with new records based on that ‘last’ timestamp attribute and remove the #infinite-scroll div case the record collection returns empty.
###### - Model (post.rb) class Post < ActiveRecord::Base def self.feed(last) self.where("created_at < ? ", last).order('created_at desc').limit(5) end end ###### - Controller (posts_controller.rb) respond_to :html, :js def index last = params[:last].blank? ? Time.now + 1.second : Time.parse(params[:last]) @posts = Post.feed(last) end ###### - View (index.js.erb) <% unless @posts.blank? %> $('.endless_scroll_inner_wrap').append("<%=escape_javascript(render :partial => 'post', :collection => @posts)%>"); $('ul').attr('last', '<%=@posts.to_a.last.created_at%>') <% else %> $('#infinite-scroll').detach(); <% end %>
Notice how we append the post partial to an .endless_scroll_inner_wrap div instead of the original UL, that’s because the endless-scroll plugin creates this new div wrapping our original div whenever we scroll said div, so we need to work around that as well. The model/controller logic is pretty dull, no need to further explain that, just make sure to always pass a future timestamp when there isn’t a parameter yet so that latest (up to the second) updates will still show.
Well, that basically covers it, the idea behind this post is to be a quick tutorial, if you’re a Rails beginner then I highly recommend you download the entire repository code and check it out (although I pasted most of it here already for explanation’s sake). Be sure to give some feedback if you have a more robust solution, this was my first try on this subject and I’m sure more experienced developers out there must have better stuff to show :)
Pingback: Endless Page Scrolling with Rails 3 and jQuery « Pedro Mateus Tavares » WB Tips
Pingback: Link dump for May 11th | The Queue Incorporated
craig said:
so the DOM will continue to grow right? say your sample application had 1million comments, would the browser really be able to handle 1million elements?
It might be interesting to improve the scroller logic such that it keeps the current viewable items (say 10 fit on the page), the last page of items and the next page of items, but removes all others from the DOM. That way if you have 1 million items you dont end up with 1million elements, but instead you might have 30 (10 from prior view/page, 10 you can see now, and 10 if you scroll to next view/page). It would still appear to be endless but all of the data wouldn’t end up loaded into the DOM.
thoughts?
pedromtavares said:
Hey Craig,
I answer your question with another question: who would have the patience to scroll down until there was a million comments? It’s the same argument as not having common pagination in the first place, most people don’t bother going to page 15, that’s an actual statistic from usability studies, or have you ever googled something and tried even the second page of results? You might have, 3 or 4 times tops, that’s my point.
So in practical terms, that’s not an issue at all because nobody will ever do that, an example that what I’m saying is true is that Twitter recently adopted endless scrolling too, and that’s considering that their web app is ALL Javascript so mass loading the DOM would seem even more relevant, and yet, they still adopt it, because they know nobody is gonna sit and scroll for 30 minutes straight.
craig said:
To play devil’s advocate, that’s sort of like Apple saying “your phone isn’t getting a signal b/c you are holding it wrong”, except here we’re saying “your browser locked up and crashed b/c you attempted to view to many results”.
However, as you mentioned it’s unlikely a user is going to go that far into the results, just thought it might be interesting to consider. Perhaps you could put in some funny message that pops up if you notice they’ve loaded some large # of results, such as, “Hey…do you really care about what’s down here?” :)
pedromtavares said:
Hey, I agree that we should cover all possibilities too and that the user should never be left hanging, but I am also quite pragmatic when it comes to solutions like these. For front-end processing to even become something to worry about, we’d have to be talking about something like 500~1000 records (or more, I have scrolled down Twitter once for like 20 minutes looking for a tweet that I should have favorited and didn’t experience any lag issues), so that takes us to at most 0.1% of chance of happening, at that case, having some warning saying ‘too many records loaded, you might start to experience some lag issues’ is a good idea.
Going back to your original solution (for the sake of argument), keeping a masked pagination between scrolls wouldn’t look quite as good if you have features such as real-time updates (most feed generators do), so if your user was in page 4 and there was a live update, what would you do, render 4 pages at once and screw the already screwed user-experience (fooling the user with a masked scroll pagination is already a usability no-no), or would you just delete 3 pages of content and show only the first one, making the user scroll all way back to page 4 (instead of just clicking the scrollbar at the bottom)?
As you can see, it’s a needless headache that would be ignored by 99.9% of your users anyway, and I’m not sure that iPhone comparison would hold the same statistics =P
drew said:
Nice tutorial. Personally I’ve always preferred the formspring.me style of having an ajax button appear at the bottom of the page which loads the next group of posts. That way you can reach the footer or something at the bottom of the page yet still have instant access to more posts.
pedromtavares said:
It’s always a matter of trade-offs. If you put a button and require a user to click it every time, you’re disrupting the natural flow of scrolling and requiring the user to go click a button, then resume the scrolling, which might be annoying to people (moving your mouse sucks). On the other hand, as you said, important footnotes will be more accessible.
What Twitter does, for example, is keep all the important stuff on top of the page so there’s nothing to look at the bottom, so you’re free to scroll.
The only situation when there’s a win-win is the one I presented in the example application, where the endless scrolling doesn’t happen on the main scroll, but on an inside scroll, so the natural flow of scrolling is kept AND you can see everything on the bottom by just moving your mouse a little and scrolling the main scroll.
J said:
Some really good ideas in here. I am planning on implementing the same thing in one of my applications shortly and I didn’t think to store the created_at date against the containing element.
In saying that, “last” isn’t a valid html attribute. You’ve used it here:
<ul class='list' last="”>
I would suggest storing it under the data-attribute like so:
<ul class='list' data-last="”>
In your jQuery:
last: $(this).data(‘last’);
$(‘ul’).data(‘last’, ”)
J said:
Some of my code above got removed when posted, but you get the gist.
pedromtavares said:
You’re absolutely correct, thanks for pointing that out :)
Rob said:
Hi Pedro,
Great post, thanks for sharing! The only thing I would warn you about is the jQuery plugin you are using puts a lot of code directly into the “scroll()” method which can have performance implications since that method gets called a LOT during scrolling.
You may want to implement a debounce or setTimeout on the scroll event as described in this blog post, written by jQuery’s creator, about Twitter’s mistake putting code in their scroll() method.
http://ejohn.org/blog/learning-from-twitter (it’s at the bottom of the blog post)
Here is a good debug plugin: https://github.com/cowboy/jquery-throttle-debounce
pedromtavares said:
Excelent comment, Rob! I’ll admit that performance was not one of my concerns regarding this plugin, but your argument makes total sense. Thanks a lot for sharing, man, great to see people are sharp about this subject :)
nathanvda said:
Excellent write-up, very useful. To combine this with `kaminari`, i set the next page number instead of the last date, and I used a data-attribute instead.
Sanjeev Mishra said:
Hey,
I have used this plugin with the changes tat you suggested for ceaseFire in case there are no more records returned. But, I have a situation, I have multiple lists on the page which have this endlessscroll. Now, in order to keep things DRY, I applied the endlessScroll on a class rather than on a div id.
Problem is that, if I do the ceasefire once, it actually stops for all the lists on the page. Do you have any idea as to why this should happen and how to get around this problem without duplicating my code!!
pedromtavares said:
Hey there,
What you could do is adopt the ‘blocker’ div I pointed out in the post for when there are no more records to scroll to, but, instead of using an ID, you could use a class like .blocker, so your ceaseFire function would look something like:
I didn’t actually test to see if this works, but I hope you get the idea.
Mischa Wasmuth said:
I just wanted to thank you for your updated endless scroll library. I’m also using Rails and jQuery but with a different approach. Your updated library did the trick. :-)