So this Google index debate is in the context of the close-up shot.

The widest perspective in this debate looks at the view from the rear view mirror of how things used to be.

How does this debate go in the context of what could be – in the future?

I’ve written here that “New Media is an opportunity to break the old media vicious cycle of using other people to make money.” http://bit.ly/4dEcOP

Importantly, I believe that by protecting the value of creating content, all participants benefit. So its great that Google is challenging the copyright issue. It means that copyright law will be re-written to protect the content creation in new media. This is critical to addressing the larger, “universe” issue – how do we capitalize on new media capabilities to build a better marketplace than “old” media. One that supports and nurtures creators of content will be much more “fertile” for everyone – from audience to media technology to programming/publishing company.

New media has yet to capitalize on the one advantage that it has over old media – the ability to sell customized products and services directly to a mass scale of consumers. So far, the new media marketplace continues to place all the risk on content creators – put it up there for free and “hope” they will come. Programming and publishing companies buy content to produce based on predictable formulas that work for mass marketing and distribution to maximize profits. Media technologies like Google are replicating the model of programming and publishing companies by “using” and exploiting content creators because their investors want to replicate the scale and efficiency of mass marketing and distribution.

Old media had to “use” others to make money. Over the air television and radio had no path to sell and collect money from consumers.

New media makes it possible to find a receptive audience, time selling messages to convert that audience to buy, and collect money. By taking advantage of this opportunity, content creators can connect with an audience who will pay for their work. Audiences who are tired of the formulaic, predictable mainstream media product, can nurture creativity. When this “tastemaker” audience so endorses a content creator’s work, mainstream publishing and programming companies will have an alternative filter to their predictable formulas.

I understand that new media companies, like Google, have been guided by their investors to mimic the old media advertising business model. They’ve come up with an efficient way to provide advertisers with a superior, performance based solution. But here’s the question. Which searches generate higher performance? Probably searches with the highest relevance to a threshold critical mass of people. What affects relevance to more people? Isn’t it the content creators, publishers, programmers? If Google shared revenues with content creators, publishers, programmers wouldn’t everyone benefit?

In my opinion, one major flaw that I believe newspapers have gotten wrong, wrong, wrong is the concept that displaying a link to their article page along with a brief snippet of their content somehow constitutes a copyright infringement. Not so – I think that falls under “brief quotes” which are considered entirely legal under copyright law. Under that concept, I can take any darn book or published material and quote a brief piece of it, for any reason, and not be considered infringing. (There may be other laws in other countries, but in most modern countries there must be some acceptable use falling under the concept of brief quotes. Or. Stuff. Falls. Apart.)

To my mind, the dicier area is what you’ve highlighted, Danny. The cached page is a lot more than a snippet quote and a slippery slope in arguing “fair use”. If an entire page is a small part of a larger work such as a lengthy book, one could maybe make the case that it’s only a small percentage of the overall whole of a work. In terms of websites, I could imagine someone arguing that displaying each page individually in a cached manner might be just a “brief quote” of an overall website.

Yet, in Newspaper terms, an entire page is often an entire article, written by a single individual, and a standalone work in itself. Further, some websites are only composed of one or a few pages — in the case of single-page websites, a cached page is an entire copy of the entire work!

I think you’re right — it looks a lot better for search engines to stop displaying cached pages (although these are so useful to me in everyday work that I’d mourn to see them go!).

I’d also like to see a way to improve DMCA copyright infringement reporting to include linking “to” (and/or monetizing those links/downloads) for illegal content. It’s one thing to enforce an “opt in” for publishers; it’s another thing to enforce a copyright violation on other websites taking advantage of the work-a-round. Unfortunately, some of the free hosted blogs on WordPress and BlogSpot are the worst offenders.

[...] things, then check out this article by the photographer (of the image on this post) Danny Sullivan: Search Engines, Permissions & Moving Forward In Copyright Battles. [...]

Mostly agree, but I don’t think the following is a problem:
“There’s no automated way to give spiders access to password-protected areas you might want them to index and list but not reveal access details to unregistered or unpaid visitors to a web site.”
In fact, there should be no such way. Listing content that’s not actually available when you click through is a frustrating and annoying experience (that sometimes happens in Google News, unfortunately).
As long as it’s freely available content, what the Google SERP does is guiding me to useful information. Fine. Bur when it’s protected content the search engine is suggesting, it simply teases me, and acts as a direct sales maker for those sites. And that’s definitely NOT fine.
Just an example:
What if I created good content and achieved the no. 1 position for some relevant, high-traffic keywords – then hide all that content behind a payment scheme? Should Google still give me the no. 1 spot? No, because the general usefulness of my content has been seriously compromised by not being freely available any more.
Actually, I wouldn’t even have earned most of my incoming links had that content been protected from the beginning.
If there was such an “automated way”, the most obvious (and deceptive) link bait would be to go free with my content for a limited period, get all the links I need to obtain high rankings, then hide it all behind a password and make people (lots of them, because I’d have huge traffic!) pay for it.
[btw, the preview option shows all the above in a single, hardly readable paragraph when in fact it's broken into several]

As a publisher more interested in getting INTO Google’s various Book and Scholar programs… I’m actually OK with what they’re doing on all counts. Also, as a publisher I’m keenly aware of what’s at stake to the publishers is MONEY. The libraries who ask Google to perform the scanning of their collections as Michigan has done are cutting significant revenues out of the publishers annual budgets. How? By bypassing the Copyright Clearance Center at the photocopier. See, every time a student copies a paper from a book… The CCC gets a fee from the library and the publisher gets a big fat check from the CCC every quarter.
Google’s online ambition stops that dead in its tracks. If you were Elsevier or Prentice Hall… you would fight a pretty big, pitched battle.

Excellent posting!
I agree with you on everything, except for one exception about the “opt-in” request for a robots.txt file. Yes, the search engine asks for that file, but if the file does not exist, then the search engine freely indexes every page it can find on the site. So, this is not exactly an “opt-in” situation. It is more of an “opt-in to opt-out” situation, since you have to know that you must create a robots.txt file in order to opt-out.
It would be much safer (with possibly fewer successful lawsuits) if the search engine left the site when the robots.txt file did not exist (or the server was unable to successfully transmit it, etc.). Not everyone knows about the robots.txt file, but they would certainly learn about in a real hurry if they wanted their pages to be included…
Brian M

Wow – that’s a great article. It seems to me that Google has tended to be a bit too free with other’s intellectual property. Their goal is admirable: to make access to all information easier. Nonetheless, the owner of the copyrighted property needs to have a say in it.
In the case of indexing a page, Google is, in my mind, in the clear: there are standardized and reasonably efficient means to restrict access. In regards to caching, although Google is perhaps invasive, there are, again, means to restrict access.
Regardless, Google’s insistence that their use of copyrighted material (such as scanning of in-copyright books) is a benefit to all needs to be addressed.