Can Google survive its blind faith in the algorithm?

Can Google survive its blind faith in the algorithm?

Takeaway: It’s been a tough year for Google search. It’s had a difficult time targeting content farms and has accidentally removed good content.

Google’s search engine is a triumph of technology. There’s no denying that.

It was the capstone that completed the initial structure of the Internet. But, the Internet is now in the midst of a dramatic remodel and it’s unclear whether Google search will get the refresh it needs to make it more appealing than ever or if it will be one of the things that gets painted over.

Google entered 2011 with two major problems that threatened the company’s immediate relevance and it’s long-term future:

1.) The search results on Google.com were becoming increasingly ineffective because they were littered with “web spam” and articles from “content farms” (sites creating faux content to turn as many ads as possible).

2.) Social media has been replacing traditional web search for many different kinds of information gathering and Google didn’t have a legitimate play in social.

The company went a long way toward addressing the second issue in July with the launch of Google+. After several high-profile social flameouts — such as Google Wave and Google Buzz — they’ve pretty much nailed it with Google+.

To be clear, we still don’t know whether Google+ will be able to win over the masses, but it has become wildly popular among tech and media professionals and it is already causing Facebook to react and make changes to buffer itself against people abandoning it for Google+. 

As huge as social media is, the even bigger challenge for Google has been the declining potency of its search engine. In recent years, Google searches have become a lot less useful and a lot more frustrating. It has become more difficult to find stuff that you know is out there — even stuff that you’ve searched for (and found) previously. Another example is pages that have posted to the web more recently. They get overpowered in the Google algorithm by older pages that have had time to accumulate more incoming links.

The big problem is SEO — search engine optimization. A whole cottage industry has arisen around helping sites optimize their pages to get ranked as highly as possible in Google. As a result, the sites that land at the top of Google search results have become more about which sites are best optimized rather than which ones have the best and most relevant content.

Even worse, whole companies have emerged whose entire purpose is to create low-quality content that is highly-optimized for Google and loaded up with ads to turn a quick buck. These “content farms” have become big business. One of them, Demand Media — which hates to be called a content farm and shuns the label — is now a public company and brags about having a close partnership with Google.

I’ll let you judge for yourself whether Demand Media is a content farm. Below are four articles from its flagship site, eHow. Are these helpful or useful? Would a site that aims to serve readers and not just serve ads publish these?

• How to Calculate Age From Birthdate
• How to Sign on to AOL
• How to Add AOL to your iPad
• How to Belch

(We’ll talk more in a moment about whether Google considers eHow a content farm.)

Recognizing the growing risks that this stuff poses to Google’s relationship with users, and ultimately its business model, the company has moved aggressively in 2011 to fix the situation. It started with a contradictory blog post in January in which Google defended the quality of its search engine as “better than it has ever been in terms of relevance” while also throwing down the gauntlet on web spam (sites that “cheat their way into higher positions in search results”) and content farms (”sites with shallow or low-quality content”).

Then, it dropped the real bombs –  a series of major updates to its search algorithm. These have been dubbed the “Panda” or “Farmer” or “Panda Farmer” updates (don’t laugh). The first one (Panda 1.0) came in February, and it obliterated search traffic to a bunch of sites, but oddly, eHow (the site most notorious for the “content farm” label) escaped unscathed.

Google eventually unleashed Panda 2.0 in April, Panda 2.1 in May, Panda 2.2 in June, and Panda 2.5 in September. According to SEO analyst Sistrix, these Panda updates eventually crushed eHow, which relied on Google search to drive most of its traffic. Despite reports of eHow’s traffic dip earlier this year, Demand Media denied that it had been hurt by the Panda updates. Then, earlier this month, the company admitted eHow’s traffic problem, although it tried to brush it off as “an internal technical issue.” The public hasn’t been fooled, as Demand’s Media’s stock has fallen precipitously.

So, Google apparently bagged its big game in the Panda hunt. The problem is that it took months to do it and a lot of algorithm trial-and-error and there was plenty of collateral damage done in the process. It’s as if Google looked at its backyard, spotted a bunch of dandelions, and instead of taking hand trimmers and going out and clipping them, Google decided to build a highly-advanced chainsaw to deal with it. The chainsaw eventually got rid of the dandelions but it also whacked some chunks out of the hedges, put some gashes into the ground, and took out part of the back fence.

Still, what this all comes down to is Google’s faith in the algorithm. Google says that it doesn’t single out sites to include or reject in Google search results. It simply builds an algorithm that systematically finds the most relevant stuff and ignores (or removes) the least relevant stuff. Google argues that this creates a fairer and more objective system, and that introducing human filtering into the system would make it biased and subjective. While that may be true, the big question is whether human intervention would make Google search more effective, and ultimately more accurate.

The problem with the algorithm (and artificial intelligence in general) is that it has no common sense or wisdom — at least not yet. Meanwhile, the systems that Google search is increasingly competing with for information discovery — social search and mobile apps — use the collective wisdom of the community or targeted experts to deliver better information more quickly than Google search, in many cases.

Despite the early success of the Google+ social experiment, the Panda updates during 2011 show that Google still believes in the algorithm above all things. The company thinks that throwing more math, PhDs, and servers at any problem is the right answer. As we’ve seen, that approach has started to fail Google in 2011. It has had a difficult time targeting content farms and it has ended up accidentally removing a bunch of useful content in the process. The big question now is whether Google can learn from this experience and change, or if it will eventually fade into becoming a fallback mechanism that people use when they can’t find the information they need from social search (asking their Twitter or Facebook friends) or a mobile app.

Four ways the Google-Motorola deal will change the tech industry

Four ways the Google-Motorola deal will change the tech industry

Takeaway: While Google’s acquisition of Motorola Mobility won’t be one of the most expensive acquisitions of this era, it will be one the tech industry’s most influential events.

Google’s $12.5 billion purchase of Motorola Mobility will go down as one of the most important events in tech during this era. Other deals, such as AT&T’s $39 billion buyout of T-Mobile USA, are worth more money, but none of them will have as big of an impact on the tech industry (and as much fallout) as the Google-Motorola deal. This is the most important acquisition in the technology space since the HP-Compaq $25 billion deal a decade ago.

Google-Motorola

Here are the top four ways that Googlerola will change the course of the tech industry.

1. The Android patent wars are over

Before Google bought Motorola, the Android ecosystem was in real danger of having innovation stymied by litigation. Microsoft, Oracle, and Apple were all bearing down on Google as well as Android partners Samsung and HTC over patent infringements. Motorola hadn’t entered the fray yet, but with its long history in the wireless business there was the potential that it could use its treasure chest of patents to pile on to its chief Android rivals, Samsung and HTC. On August 11, my ZDNet colleague James Kendrick posted, “If Motorola turns its patents on other Android phone makers the platform will implode.” And, that was on top of the overall intellectual property issues with Android itself, which affects all of the Android device makers. The problem for Google was that it didn’t have enough mobile patents to fight back. That’s the way these things usually work. One big company typically says to another big company, “Yeah, we might be infringing you there, but you’re infringing us over here” and then it turns into a draw. With Motorola’s 17,000 patents on its side, Google has essentially put an end to the Android patent wars. There will still be some final skirmishes, but don’t expect much carnage.

2. Vertical integration has won

While Google is pledging to keep Android an open ecosystem and claiming that it will run Motorola as a separate business, it’s pretty clear that Google also wants to have the option of producing its own hardware devices so that it can build prototypes, concept hardware, and leading edge devices to demonstrate its vision and point its ecosystem partners in the right direction. Google wanted to do this with the Nexus One smartphone and we also saw Google’s hardware itch in the CR-48 laptop running Chrome OS. Of course, Google didn’t have the expertise or infrastructure in place to handle the hardware business. With the Motorola Mobility acquisition, it will add over 19,000 new employees with supply chain, customer service, and hardware development skills. When Google wants to do its next leading edge Android device like the Nexus One, Nexus S, or Motorola Xoom, we have to assume that it’s going to use its new hardware division to build it so that it can deliver exactly the device it wants and get it to market much more quickly. With Apple’s continued success in mobile, BlackBerry’s large (albeit fading) market share, HP’s new hardware/software unification with WebOS, and now the Google-Motorola deal, it’s becoming clear that vertical integration is winning in mobile. Going forward, look for the latest, greatest, high-end devices to all be vertically integrated, while many of the low-cost, copy-cat devices will come to the market later and be made by mass market manufacturers like Samsung.

3. Mobile consolidation has begun

Over the past couple years, the arrival of new mobile platforms and the expansion of mobile vendors have given buyers lots of new choices in smartphones and now tablets. However, even in a fast-growing market like mobile, the good times can’t last forever. In 2011, we’ve already seen BlackBerry and Nokia drastically losing momentum, Windows Phone 7 and WebOS struggling to gain market share, and Android and Apple increasingly hogging the spotlight. Even within the Android ecosystem itself, there have been lots of new upstarts recently, including LG, Lenovo, Acer, and ASUS. All of them have been grasping for a piece of the expanding Android market, which has been dominated by the big three — HTC, Samsung, and Motorola. However, leading up to the Google deal, Motorola was the only one of the Android vendors that lost market share in the smartphone market in Q2. Obviously, that’s likely to change if and when Motorola morphs into the Google-branded Android devices. Nevertheless, Motorola’s Q2 struggles are a sign that the Android market itself is already beginning to whittle down to fewer big players.

4. Google has to grow up

As a company, Google is only a little over a decade old. Despite its recent kerfuffles with government regulators and its dust-up with China, the company has lived a bit of an idyllic, Peter Pan existence. Its offices are like college campuses with free food, free transportation, and free personal services (cleaners, barbers, etc). Its employees are loosely organized, don’t have to deal with a bunch of overbearing middle managers and bean counters (in most cases), and even get the ability to use work time to dabble with some of their own pet projects. Because Google’s search engine has been such a major cash cow, it has given the company freedom to hire lots of engineers and computer scientists and loosely organize them in this unique environment. However, with search under greater pressure than ever from the social web, it could finally be time for Google to grow up and act like an adult company that has to closely manage expenses and account for the value that each of its employees brings to the organization. The Motorola acquisition could hasten the process, since it will add over 19,000 employees to a Google that currently has 29,000, and Motorola is a much more established company with traditional organizational standards. Of course, Google will talk about wanting to maintain its startup-like culture, but it will be interesting to watch and see if Motorola influences Google to become more of an accountable, grown-up company.

*********************************

Scroogle: Adding privacy to Google Search

Takeaway: Google Search is an amazing tool. Even so, to many, it has a dark side. Scroogle may be able to help.

Over the years, I’ve witnessed–from a safe distance–highly-charged debates about search behemoths like Google. The topic most often discussed is whether or not they retain too much Personally Identifiable Information (PII) for too long. Valuable lessons surfaced from those frank discussions, many important enough for me to write about.

Another place where I have gleaned similar information has been in the comment sections of the articles I just mentioned. One example is my introduction to Scroogle.

My first impression was: What an odd name. I didn’t think much more of it. Then a colleague gave his middle-finger explanation of the term. “Oh,” was all naive me could say, “You really think so?”

Scroogle, what is it?

Now I had to find out about Scroogle. First thing that caught my eye:

“Every day Scroogle crumbles 350,000 cookies and blocks a million ads.”

Next thing I noticed, Scroogle does not:

• Pass cookies on.
• Keep search-term records.
• Retain access logs for more than 48 hours.

The website calls Scroogle a scraper. Being from Minnesota, I have this image of a scraper and it is not Scroogle.

Actually, after some study, referring to it as a scraper does make sense. The pertinent search results are “scraped” from Google’s response to the search query. And only that information, no cookies or additional requests, get back to the client’s web browser.

The following slide depicts the steps involved (courtesy of Scroogle):

Behind the scene

The process is simple. You enter your search request in the web browser, like normal. It is sent to Scroogle via a SSL connection — more on that later. Scroogle replaces all your identifying information with that of Scroogle. The search request is forwarded to Google. Google records the IP address and search information issued by Scroogle.

Google then replies with a cookie and the search results. Scroogle sanitizes the data, sending only the search results back to you. Below are the search results for ice scraper using Google:

Next are the results using Scroogle:

Scroogle, the plugin

The website calls Scroogle a browser plugin. Simple enough to implement, but I’d like to expand on the minimal help offered by the website:

• Firefox: This link is to the Firefox add-on. All that is required is to click on the Add-on button.
• Internet Explorer: Microsoft set up Internet Explorer to ask for the desired search engine. Details are at this link. All that is required is to enter http://www.scroogle.org/cgi-bin/nbbw.cgi?Gw=TEST where it asks.
• Opera: Click on the following: Tools/Preferences/Search/Add. Pick a new keyword “example” and use http://www.scroogle.org/cgi-bin/nbbw.cgi?Gw=%s as the address.
• Chrome: Click on Wrench/Options/Default Search Manage/Add. Then paste https://ssl.scroogle.org/cgi-bin/nbbwssl.cgi?Gw=%s where an URL is requested.

If you prefer not to alter the current configuration of your web browser, or are using a computer other than your own, Scroogle has a webpage similar to Google, where you can enter search terms.

Back to SSL

The Scroogle website points out why the creators decided to use SSL connections:

“For Scroogle, SSL is used to hide your search terms from anyone who might be monitoring traffic between your browser and Scroogle’s servers. This encryption happens when you send your search terms to Scroogle, and it also happens when Scroogle sends the results of your search back to you.”

The SSL webpage points out another advantage that I was not aware of:

“When the Scroogle results come back from an SSL search, and you click on any of the links shown on that secure page, there is another advantage. SSL does not allow the browser to record the address where that secure page came from and attaches it to any outgoing non-SSL links on that page. Normally all browsers do this and it’s called the “referrer” address.

Using SSL blanks out this referrer, so that any non-SSL site you click on from a Scroogle SSL page won’t know that you arrived at their site from Scroogle. The referrer will be blank, and your log entry at that site will look like any of the hundreds of bots that crawl the web all day and night with similar blank referrers.”

I did not know that until now.

That said, do not let the use of SSL connections lure you into a false sense of security. SSL may or may not be in play after you click on one of the returned search links. It depends on whether the web server advertised in the link is using SSL or not.

Both use SSL

Google also has the option to use SSL. And, Google makes the same claim on how encryption prevents third parties from intercepting transmissions between the user’s computer and Google Search web servers.

My immediate thought: It would be cool if the Scroogle servers talking to Google Search would use their SSL connection. I shot off an email to Scroogle and Daniel Brandt, Founder and President of Scroogle, offered this:

“No, the connection between my servers and Google does not use SSL.

There are two reasons for this:

• The search terms for that hop are carried by the IP address of my server, and the only way they can be associated with the searcher’s IP address would be if someone hacked into my dedicated servers and read my logs. And they’d have to be quick about it, because I don’t keep any logs longer than 48 hours. I’m the only one with access to my servers.

• I do not use DNS to do a lookup of www.google.com. Instead, I randomly select one of their static IP addresses for www.google.com (they have thousands). As you may know, https initiation requires a handshake that certifies that the domain name belongs to the IP address. Since I’m not using “www.google.com” at all, I cannot initiate an https session with Google.”

That makes sense to me. Thank you for clearing that up, Daniel.

Quality of SSL connection

I just happen to be researching a new Comodo website, SSL Analyzer. It is a free web-based scanning tool that checks the security of a web server providing SSL connections.

Included in the summary is information about the certificate and digital signature. Also included, is a list of security protocols and encryption suites supported by the web server.

SSL Analyzer uses the following designations to highlight problems:

• Red: Problem that needs immediate attention.
• Amber: Potential issue that needs evaluation.

With so much emphasis being placed on SSL connections, I thought, why not test them? Here are the results for Scroogle and the results for Google Search. You can see that both have issues. I am not sure I would consider them show-stoppers, but it is something to think about.

Bottom line

Now comes the hard part. After all is said and done, it ends up being a matter of trust. If using Google Search is important, but you are not sure about trusting Google, you may want to think about Scroogle.

The top 10 features missing from Google Plus

The top 10 features missing from Google Plus

Takeaway: Google+ has captured the attention and the imagination of the technology world. But, it’s easy to forget that it’s still in closed beta. Here are the top 10 things that need to be fixed or added.

Google+'s Features

There’s a lot to like about Google+ and it has the potential to make a major impact on the future of the Internet. After just two weeks in closed beta, it already has 10 million users. Still, it’s far from perfect. I’ve put together my list of the top 10 things Google needs to fix or add in Google+. Take a look at the list and then jump into the discussion and argue with me by adding the fixes that you think deserve more attention in Google+.

Google+ Logo

1) Let us mute someone from the Stream
On Google+ it is a lot easier to find friends and people to follow than any social network that has been built so far. And, with Circles, you can divided them into groups and then easily jump between the various streams of your Circles. However, there is also the big “Stream,” which aggregates all of the people you have in Circles and this is the default view you see on Google+. The one feature missing here is the ability to mute a person from the Stream (while still being able to see their updates in their Circle). You can mute individual posts from the Stream, but you can’t currently mute a person. This is badly needed so that you can stay connected to interesting people but not have the overly chatty people monopolize your Stream.

2) Show list of my +1 items from Google+
The +1 button allows you to gives the thumbs up to really good Google+ posts and updates. However, this should also work like a list of favorites or bookmarks. Right now, there’s no way to see a list of the things where I have clicked +1. If I go to my profile there is a +1 tab, but that’s the list of external items (from web sites or Google search results) where I’ve clicked +1. The items from Google+ itself need to be added to this list.

3) Fix the share and re-share issue
One of the stickiest issues Google needs to figure out is the Share functionality. If you’re familiar with Twitter, this is like a Retweet (RT). However, when you share a post on Google+, it removes all of the comments and +1s, allows you to add your own comments above the post, and then your followers can add their own plusses and comments. That can be pretty cool, except when a bunch of the people you follow all share and re-share the same post. With the approach Google has taken to sharing, there may not be an easy answer, but something will need to be done to sort this out, at least for stuff that gets shared more than 2-3 times in your stream.

4) Let us sort the stream by raw timeline
By default, the big Stream (and the Circle streams as well) are sorted by relevance and popularity, based on the number of +1 votes and shares, so that the most interesting stuff rises to the top of the Stream. However, Google should also give us the option to sort the stream based solely on timestamp, so that we can see the stuff from people who post interesting things but don’t have as many followers to buoy their posts.

5) Allow comments to be threaded
This is an issue of intense debate, but I think Google should allow threaded commenting on Google+ so that people can comment on and respond to comments, and not just the original post. That would make the threads a lot easier to follow when they get a lot of comments. And, comments on comments could be collapsed by default and users could simply click a plus sign to expand and view them. However, the threading would only need to go three layers deep to allow a response and a counterpoint.

6) Add more functionality to mobile
It’s impressive that Google had its Android app for Google+ ready to download the moment that it launched the “Field Trial” of the new service, and shortly thereafter it submitted an iPhone/iPad app to Apple for approval in the App Store. Even better, the Android app for Google+ is veyr well done. However, it’s not perfect and it could make the mobile Google+ experience a lot better by adding key functionality — e.g. the ability to +1 a comment, the ability to join a hangout, the ability to easily flip between the big stream and circle streams, etc. While they’re at it, Google should add more core functionality to its HTML app as well. That would be a great way to drive more participation and get a jump on Facebook, which still doesn’t have a great mobile experience.

7) Open it up to Google Apps users
In order to get into the Google+ beta you need a Gmail address (or a Google Account). It does not currently work for the Google Apps domains, which are business accounts where the company is using a corporate version of Gmail, Google Calendar, Google Docs, and other Google web apps. Google has stated that Google+ pages for businesses and brands are coming soon. Hopefully, Google+ for Google Apps users won’t be far behind. Since some people may end up having separate Google+ accounts for personal (Gmail) and business (Google Apps) use, Google should also consider options for letting those users log in to both accounts from different tabs in the Chrome web browser.

8) Integrate private messaging
One of the biggest things Google+ is missing compared to rivals Facebook and Twitter is the ability to send a private message to a mutual contact. Sure, there are a few workarounds and hacks that let you do it, but Google needs to make this part of the product’s primary functionality. It also wouldn’t hurt to integrate GoogleTalk (instant messaging) as well.

9) Set up verified accounts
Since there are already celebrities showing up on Google+ — and a lot of people who are impersonating celebrities — Google needs to set up something similar to Twitter’s Verified Accounts. Just do a search for “Mark Zuckerberg” or “Lady Gaga” on Google+ and take a look at how many accounts there are. There are also plenty of sneaky imposters, like the person who pretended to be Apple’s Jony Ive (the account has been deleted).

10) Show list of interactions with each user
Another useful feature that Google should add is the ability to go to a user’s profile page and see all of that person’s interactions with you — their +1s and comments on your posts, as well as your +1s and comments on their posts. This would help figure out if you should add a person to your Circles, and if so, which Circles you should put them in or add them to.

************************************