Mobile Content Transformation

Tom picked out one particularly juicy tidbit from Novarra, and James Pearce points to the dotMobi response to the issue in his comments. It’s great! I might start using it as my example of why participating in W3C is useless, it’s a stellar performance.

I realise that what I am proposing puts the onus on servers that have more than one representation to do something that they don’t today, and I realize also that I am suggesting that transcoding proxies can modify the user agent string.

Yea, that’s not going to work. What’s really amazing is the signal to noise ratio. So many emails and they can really be summarized in just a few points:

  1. Mobile developers, we don’t care what you think.
  2. We’re changing the user agent anyway.
  3. We’re hiding the ways to request the original user agent behind a convoluted process so that we can call new developers dumb when they don’t understand it. Cause the way to solve problems that are difficult is to make them more complex. It makes us feel good about ourselves.
  4. We still don’t care what you think.

Go W3C! Well done.

This entry was posted in ThisIsMobility. Bookmark the permalink.

15 Responses to Mobile Content Transformation

  1. I don’t see how this exemplifies why W3C would be useless, although I’m obviously biased in this regard.

    The way I see it, W3C offers a public discussion space where the different parties are able to expose their position, discuss the problem they may create, and they to explore better ways – on which the public at large will be invited to comment later on.

    How is that worse than having companies doing their work on their own, keeping their ways as is without taking any input into account?

  2. Sean Owen says:

    I don’t follow the logic here. If you don’t like what Novarra’s proxy is doing, don’t you want ways to say “get out of the way, don’t transcode”? I think that’s the thrust of Jo’s post, that there are existing, standard ways to do this “on the books.” I would have thought that people really concerned about content adaptation would already know about these, be pushing these, and be glad to see stuff like “Cache-Control: no-transform” highlighted as a way to get a transcoder to redirect the browser directly to the target site.

    Whether you think the Novarra proxy should be there at all is a different story — it should absolutely respect a sites wishes regarding transformation.

    I understand some people / companies depended upon a transcoding proxy not being in the middle, and this change broke their technology. The suddenness and lack of notice is rightfully upsetting. I don’t know that I have seen any good argument that the User-Agent behavior is *wrong*, just not what people expected. FWIW Google’s transcoder should be taking heat too by all rights; it has always changed the User-Agent header to (correctly) say “you are talking to a transcoder, my friend.”

    … and finally, come on, do you really think anyone in this biz really sets out thinking “we don’t care what you think?” I personally really think the W3C and dotMobi and that whole posse (which includes me/Google, who are no fools in this space) are trying to be the ‘good guys’. Er, maybe I misunderstood your post and you’re actually trying a swipe at Novarra.

    Again for what it’s worth — as a result of the discussion Jo + dotMobi have brought up, we are about to roll out a change to GWT that should make it deal more properly with no-transform directives. That’s helping actual developers and sites work better — not a wasted conversation at all. The bitch-fest I see all over the internet about VF + Novarra isn’t doing much though…

  3. Nigel C says:

    Sean,

    On the question whether anyone in the biz really sets out thinking “we don’t care what you think”… one only has to look at Novarra’s Draft of the W3C Content Transformation guideline:

    http://lists.w3.org/Archives/Public/public-bpwg-ct/2007Sep/0029.html

    See this gem right here:

    “2.3.1.7 A content transformation server can do a better job of following
    mobile best practices
    The “Mobile Web Best Practices 1.0″ W3C Proposed Recommendation [1]
    contains many recommendations for authoring content that is intended for
    viewing on a mobile device. A well-designed content transformation
    server can do a better job of following the mobile best practices than a
    human author, especially when taking into account the capabilities of
    the many different mobile devices. The result will be a more
    consistent, uniform experience.”

    And this:

    “Content transformation servers typically want the origin server to send
    the desktop version of the site since the desktop version is usually
    more functional.”

  4. James Pearce says:

    Just to make the full house, I’ll chip in.

    The verbosity of a typical W3C mail thread shouldn’t mask the fact that its constituents are desperately trying to do the right thing.

    In our (dotMobi’s) case, we think transcoding is a short-term necessity to cover for the fact that consumers can’t yet find a critical mass of sites made for mobile.

    So we’re trying to point out that our developers (who *do* build sites made for mobile) can make two tiny changes to their sites which declare their sites untransformable.

    About 30 seconds worth of configuration I would say. It’s quicker to add a cache-control directive to a site than to go to the effort of registering it on a single operator’s white list, let alone all all the other similar platforms that are surely on their way.

    And we’re going to continue to make this point. Increasingly punchily.

    At that point, the emphasis will then be on ensuring that the aforementioned transcoders do what they’re told. Google’s ahead of the curve, as you can see.

    All good.

    So to return to the point. Basically, W3C appears to be painfully aware of the pain that developers are feeling today – and is trying to do something about it… albeit in its own way.

    When it looks like things are only going to get worse, I think that’s when standards bodies start to earn their (humble) fees :-)

  5. David Harper says:

    @ Sean.

    With all due respect…

    As mentioned here:

    How Vodafone and Novarra Killed Mobile Commerce
    http://www.seoprinciple.com/how-vodafone-and-novarra-killed-mobile-commerce/20/

    and here:

    Vodafone UK, Doing A Lovely Job Of Supporting The Mobile Web By Breaking It
    http://mobhappy.com/blog1/2007/06/06/vodafone-uk-doing-a-lovely-job-of-supporting-the-mobile-web-by-breaking-it/

    This issue has been affecting mobile companies since at least June. While this might not seem like a burden to people at large companies I can assure you the impact on small ones is significant.

    Sometimes a bit of well deserved “Name & Shame” works wonders when big co ignores the little people. Re: the comment you have been sprinkling about “The bitch-fest I see all over the internet about VF + Novarra isn’t doing much though…”

    …apparently has had EXACTLY the affect that was intended.

    Vodafone paid attention and the right people are talking together and sorting it out.

    Note: I really like Sean and respect the smart, good work he has contributed towards mobileOK Best Practices.

    Re: “be glad to see stuff like “Cache-Control: no-transform” highlighted as a way to get a transcoder to redirect the browser directly to the target site.” and “don’t you want ways to say “get out of the way, don’t transcode”? ” and “Whether you think the Novarra proxy should be there at all is a different story — it should absolutely respect a sites wishes regarding transformation”

    Yes. …and in this case or others that label a site “No Transform” the User-Agent String needs to be passed intact (which I understand is the case) so the correct content, version, templates, experience etc. can be delivered.

    Cheers,
    David Harper
    Founder, Winksite

  6. miker says:

    I was mostly swiping at Novarra, but I also don’t think the proposed solution is very good. As far as a good reason not to change the User Agent by default, because there are a lot of sites (and tools that sites are built on top of) that don’t look for the alternative header. I spend a lot of time talking to people who publish mobile websites. They struggle and muddle their way through existing tools, vague and frequently conflicting advice, and a mishmash of protocols and markups that even the most technically adept usually shy away from. They’ve managed to get their site up and working anyway. Now you’re going to force them to go back in and change something when really there’s also a way to provide transcoding without making them change something. Sure, it’s “only looking at a different header”. But not everyone out there publishing mobile content is doing it cause they love technology. Many of them are doing it because they love whatever topic it is they’re publishing about, and they tolerate the technical hurdles in order to reach a mobile audience. Why put something else in their way, no matter how small?

  7. James Pearce says:

    Now that I fully agree with ;-)

  8. Mike,
    the link you provided is a suggested text, I believe. Nothing that has been approved, it’s food for thought. You may agree or disagree and I believe you disagree. ;)

    Once we have agreed that:
    a) there’s a number of sites providing cool mobile content that should not be touched
    b) there are many sites that do not provide a decent mobile experience

    How do we make the two sides happy?

  9. Sean Owen says:

    To be clear, the substance of my recommendation (and the one from Jo you linked to) isn’t “just read this other header.” I agree you need the original User-Agent header if you’re doing content adaptation — really, you need to speak directly with the phone. That’s why I think this issue is about emphasizing doing the right things with Cache-Control, and not really about User-Agent. That is, I don’t see why one would be so up in arms about adapting content when it’s just going to a transcoder — the issue is getting it out of the way by telling it to 302 redirect to the target page.

    This isn’t somehow changing the rules; proxies and these headers have been around for a while. Yeah I agree the mobile technology ecosystem is complex stuff and you can’t just throw up a page and get it right. That’s why I’m most happy to see timely, constructive discussions coming out of the W3C at the moment that offer the ‘right’ solution for transcoders and developers, a way forward. I know one transcoder that’s going to improve its behavior in this regard as a result.

  10. Sean Owen says:

    @Nigel I can see that the first quote is a little incendiary. They are right insofar as most mobile sites out there can’t get within the ballpark of even “valid markup” while a transcoder’s output probably will be. But no, a customized site by a knowledgeable author can only be better. The other quote is just saying that a transcoder would prefer to transcode a desktop site since there’s more to it.

    And did I not just see that Novarra’s proxy will stop transcoding sites with “mobile” or “wap” and so on in the domain? I think that’s great insofar as it’s addressing the underlying issue: not transcoding sites that don’t want or need it. Again, the issue is not that people should actually demand a transcoder say it’s a phone, since it isn’t, and would lead to other bad behavior.

    Hmm… who started everyone on this chant of “User-Agent! I want my User-Agent!” anyway? glad to see the right things are happening anyway.

  11. Jo Rabin says:

    Mike

    Late to the party, I just thought I would point out that the selective quote from my post to W3C Content Transformation List doesn’t quite convey the full sense of what I was saying. So for completeness:

    I realise that what I am proposing puts the onus on servers that have more than one representation to do something that they don’t today, and I realize also that I am suggesting that transcoding proxies can modify the user agent string. In light of recent intemperate discussion, on this and other lists, let me repeat that what I mean is that they can do this only in order to avoid a blocking response. If, in response to a request, they get a Vary: User-Agent or * they MUST re-request the URI with the User-Agent they received and remember that setting for that URI (using a mechanism similar to authentication realms in which user agents are expected to remember the URI path and re-present authentication credentials without prompting the user for URIs in that path).

    I agree with your subsequent post and the obstacles to creating content and how difficult this is even for quite technically literate people. But that is a separate subject, in my view:

    What do we want? Decent usable tools! When do we want them? Now!

    But first, let’s stop content being mucked up whether it was created with decent tools or not.

    Jo

  12. miker says:

    @Jo

    I just don’t see how that order of operations works out:

    “If, in response to a request, they get a Vary: User-Agent or * they MUST re-request the URI with the User-Agent they received and remember that setting for that URI (using a mechanism similar to authentication realms in which user agents are expected to remember the URI path and re-present authentication credentials without prompting the user for URIs in that path).”

    If the transcoder has requested the page without the original UA then the server will most likely return a desktop formatted page (if we’re talking about current behavior of most sites that have a desktop version and a mobile version).

    Why aren’t we making existing desktop only sites opt into trascoding instead of asking the people who have mobile versions to opt out? It’s much easier in that direction, there’s only one format and if it comes back with the “Yea-Sure-Screw-With-This” header then you can do whatever you want. Instead of multiple requests back and forth to the server and modifying Vary headers and sending cache control and requesting that the original user agent be returned with the next request.

    It doesn’t magically enable all the web for mobile usage, but spend some time browsing around through a transcoder and I’m pretty sure you’ll think there’s nothing magical about it. It’s better than nothing, I like the whole trascoding idea. But I think it’s overstepping bounds, especially when an entire carrier starts sending all their traffic through a transcoding system (I consider Google in a separate class here, they don’t automatically get a swipe at whatever any user puts in the location box of their mobile browser) to hide the original user agent in the outbound request.

  13. Jo Rabin says:

    Mike

    I agree with your sentiment and only wish that I believed it could work that way. At risk of being flamed for this view, and with a bunch of caveats below, I think we need to come up with something that allows transforming proxies to transform sites that know nothing about them, like the many sites that haven’t been touched in years.

    Caveats: I’m not sure quite what the benefit of bringing stale sites to an adoring mobile audience is, not am I sure that a transformed experience of them is worth having either. However, we need to accept that different players in the mobile value chain have different perspectives and someone needs to blink in order to make progress.

    Since in general (but see http://site.mobi), as we just mentioned, you have to have a degree of tech savvy, regrettably, to build a mobile site and since these are far fewer right now, practicality says to me that the change has to be that way round. Add to that the fact that some kinds of broken markup and content actually crash the mobile devices (though that is getting better) you are really not doing the end user much of a favour by saying that everthing should be sent through untransformed, unless the site says otherwise.

    To build the mobile ecosystem we need to encourage users. We can do that by on the one hand saving them from the worst effects of broken web sites, and on the other hand by building great experiences for them. If we build great experiences for them we need a way specifically to tell third parties to leave those experiences alone.

    Jo

  14. Pingback: MobHappy » Blog Archive » No Wonder Novarra Is Mucking Things Up So Badly

  15. Pingback: ipod resources

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">