We got a demo of a fancy new web-based software tool -- project management, task tracking, bug tracking, collaboration, Agile, Extreme Programming, a singing, dancing, iterating revue. I know a little something about the tradeoffs that creators of collaboration tools have to make. So I could ask pesky questions like:

 

• Do you host it, or do we have to? If the latter, what sort of server, database, etc. do you recommend we set up? On average, how long does it take a customer of yours to set that up?
  
  
• What's customizable and what's one-size-fits-all? For example, does using your application require that the user fill out certain fields for every task, and does the administrator configure that? On average, how long does it take a customer of yours to choose and configure the customizable parts? (You can't have both off-the-shelf and customized.)
  
  
• What's performance like? (Unfortunately, to check their reply, you have to play around with an instance or installation that has a lot of data in it. See if the vendor has a trial version that comes with sample projects.)
  
  
• What browsers do you support? (If you have people in your organization dedicated to Opera or Konqueror or iCab, see if the creators of this app have even heard of them.) What versions? (Watch out for IE7 and Vista issues!)
  
  
• If someone assigns a task to me, how do I find out? Email? RSS notification? Or do I have to keep this web application open all the time to find out whether I'm the bottleneck keeping us from meeting our commitments?!
  
  
• What's not so great about your application? Related: When do you recommend we not use your application? (I answered this honestly when I worked sales, so I want a straight answer when I ask it. Salon.com wasn't for those who can't stand reading on a screen, or have no free time. FogBugz wasn't for secretive organizations with six levels of security, or time-rich, money-poor hackers. You'll gain credibility and customer satisfaction if you're upfront about who's not your target market and why. If you think you've made the One True collaboration tool, perfect for all niches, then you need to get honest with yourself about the architectural choices embedded in the product you're selling.)
  
  
• What do I lose in switching to your tool? (Example: you'll lose the tactile intuition of the index cards you use for Agile planning. You'll lose the ability to destroy code or documentation without leaving a trace. But that's worth it because...)
  
  
• What decisions does your tool make easier?
  
  
• Does your product integrate with...? Twitter, SharePoint, Subversion, FogBugz, AIM, Outlook, Microsoft Word's Track Changes, iPhones, BlackBerries, Asterisk, Basecamp, TickSpot, Crystal Reports, Time Machine? And if the vendor says yes: How does the integration work? (Example: If half your staff have iPhones, then check whether "integration" means "the site works on iPhones" or "iPhone users can use a special iPhone-optimized version of the site." If you use FogBugz, there's a difference between "we'll email new issues into your FogBugz installation" and "the site makes extensive use of the FogBugz API.")
  
  
• How do I get ALL my data OUT? (Best answer: "You always have access to all your data in [common, open-protocol format] and can export and archive it at any time by clicking this link." If you don't want to act like you're already thinking about lock-in and the endgame, just say you need regular archives as part of your backup system.)
  
  
• What's your upgrade history? When was the last upgrade, and when is the next one? (Check this, the release notes, and whether an ecology of plugins exists around the tool.)

I don't think the vendor liked that I was asking these questions. I got the nervous-laughter/"That's a very good question" combo twice. Good sign.

I can come up with more such questions for vendors in case anyone feels like lengthening their checklists. Share yours in the comments.