Concerns of Real Search Engine Developers

Concerns of Real Search Engine Developers

Udacity updates the course material from time to time. A unifying thread that ran through the course was development of a web search engine. Some while after I'd completed working through unit 6, they added a set of Q&A sessions. Q&A 6-6 is here: http://www.udacity.com/view#Course/cs101/CourseRev/apr2012/Unit/676073/Nugget/674085

Q&A 6-6 is a discussion of the current concerns of real search engine developers.

After viewing that clip, it occurred to me that a major concern of real search engine developers hadn't been included. That led to an exchange of e-mail with the professor, and at his request, I posted that exchange to the CS101 forum.

The original forum thread is here: http://forums.udacity.com/questions/2025563/concerns-of-real-search-engine-developers#cs101

Here is what I posted:

I sent some e-mail recently to Professor Evans:

I just stumbled upon Q&A 6. I guess you guys added it since I'd already completed chapter 6. Q&A 6-6 about what are the challenges for search engines these days could use a little expansion. A major challenge that I don't think you mentioned is how to keep the search engine attractively useful while also figuring out how to make a buck. A recent discussion on facebook started out with a friend mentioning he'd recently purchased some fresh turmeric root and was wondering what to do with it. I suggested he just Google for:

turmeric root recipe

and pick what he wanted from there. I even pointed to a specific page that was early in the search results and looked like a good answer to his question. But the thread degenerated into a lot of gripes about how Google clutters things up with ads, and wishing that someone would do better. Not sure if Facebook will block you from accessing it, but here's the URL for that thread: http://www.facebook.com/barryshein/posts/10151255645449313 That discussion is now up to 54 replies. Clearly a topic about which people have strong opinions. (To be fair, Barry's friends probably are more geek- inclined than a random sample of the population would be. As evidence I offer 2 examples of search terms mentioned in the gripes:

microinverters

double walled heat shrink tubing

).

Prof. Evans graciously replied:

Thanks, Drew. Yes, there is definitely a lot more that goes into this, and a lot of things lead to conflicts between commercial interests and good search results. DuckDuckGo seems to do pretty well on these searches.

Happy to have discussion on this in our forums.

Best,

--- Dave

I'm happy to open the matter up to forum discussion as he suggests. How are the search engines doing, in your opinion, at balancing the interests of providing good search results (to keep us users happy) vs. making a buck to keep the lights on (and the stock-holders happy)?

Addendum

The posting stirred up much less controversy on the forum than I'd expected. Sentiment among the CS101 students who bothered to respond seemed to be that Google was doing a decent job of balancing promotional info and search results, and that DuckDuckGo was a comparatively weaker search engine.