total-impact blog

Open impact metrics need #openaccess. Please sign.

Something exciting is going on.  A petition for increased access to the scientific literature is gathering steam.  If it gets 25k signatures in 30 days — and it looks like it will get many more — the proposal will go to Obama’s desk for integration into policy.

Total-Impact urges you to sign this petition and share it with others.  We have :)

Improved access to the research literature is *essential* if we want innovative systems to track the impact of scholarly research products within the scholarly ecosystem.  

As far as we know, there is only one cross-publisher open computer-accessible source for citations: PubMed Central. And the only cross-publisher search of full text that can be reused by computer programs? Comes from PubMed Central.  PubMed Central is awesome, but it only has NIH-funded biomedical literature. Scholarship needs these resources for all research literature.  This petition is an important step.

Please go sign the petition and spread the word.  #altmetrics #OAMonday #openaccess #theFutureIsComing

Durham Smoffice application

Thanks to Alert Community Member Hilmar Lapp, we found out about a contest for startups to win the World’s Smallest Office (smoffice, get it?): basically a nook in a coffee shop. Rather more enticingly, it also comes with six months of a condo in downtown Durham, which has a growing startup scene.

Jason lives and goes to school near Durham, and plus we like the kind of wacky thinking that comes up with this sort of contest so we applied.  Here’s the one-minute smoffice video, in which I sell TI from inside a boiler room, dryer, car trunk, and my fridge. 

Here’s the one-page business plan we submitted—bear in mind that in keeping with our agile approach, this is a business plan, but not the business plan. It’s certain to change, maybe quite radially, as we continue to adapt in response to user feedback and our own evolving ideas.

We find out on the 13th. Feel free to vote up our video on the smoffice Facebook page if you’re into that sort of thing.

total-impact awarded $125k Sloan grant!

We just heard: total-impact has been awarded $125k by the Sloan Foundation! What’s this mean for users?  By April 1, 2013, we plan to hit important milestones in three areas:

Product:

• addition of over a dozen new information sources to total-impact.org, particularly data repositories
• 60 github watchers, 20 forks
• substantial innovation in user interfaces for and visualizations of altmetric data

Use:

• 50k visits to total-impact.org, 30k unique visitors
• at least 100 scholars embedding or linking to TI reports on their CV
• at least 25 TI reports included in annual review or tenure & promotion packages
• 15 publishers/repositories embedding total-impact data on articles/datasets
• 5 in-process or published research studies based on TI data

Sustainability: A sustainable business plan and organizational model for a mission‐driven TI organization

For more detail, see the grant proposal.

We are so excited! Thanks to Josh Greenberg, program director.  You won’t regret it :) As always, let us know if you’ve got thoughts or ideas on how we can best make these goals happen. Now let’s go change the world!

follow along as we rearchitect

Total-impact has outgrown its baby teeth: we are rearchitecting the codebase.  The goal is a robust and scaleable framework that will take us through the next phase of rapid growth.

The new codebase will have a clean api, a webapp that uses the api directly, data storage at the item level, a history of metric values over time, and queues to facilitate timeliness and scalability.  It is being built from the ground up with good logging, error-handling, and documentation… aspects that aren’t always at the top of the hackathon agenda :)

The new codebase is written in Python rather than PHP.  This change wasn’t taken lightly: changing programming languages is a Classic Blunder after all.  That said, others have done it successfully, and Python appears to be the favourite programming language at Hacker News, so we’re confident it is the right move.

Without further ado, here are the new code repositories at GitHub.  Works in progress… stay tuned!

agile meets distributed open source development

Total-impact will always be open-source.  It is a pretty standard OSS project in its early stages: its core developers are geographically distributed and contributions have been fueled by enthusiasm rather than paycheques.  Producing Open Source Software gives a great overview of standard processes for these sorts of projects.

At the same time, Jason and I (Heather) are sold on the principles behind agile development: iteration, adaptation, tight feedback loops, simplicity.  

OSS and agile methodologies have many similarities, but some differences.  In particular, agile development is practiced most often by co-located, dedicated-coding teams.  As a result, we’ve been rolling our own process a bit.  It is working well: it feels good, we aren’t spending too much time on process, and we recognize and change things when they aren’t working. 

For example: we were keeping our sprint backlog in a google spreadsheet. Last sprint we moved to tracking sprint items as GitHub issues.  Although this makes it harder to do time estimation, it is easier to integrate into our workflows… currently a win.  

Here is what our development process looks like right now:

• two-week sprints.  We practice a little more flexibility in pre-determined scope than Scrum agile.
• developer conversation take place openly on the newly-formed total-impact-dev google group
• weekly Skype calls with active developers (Richard, Mark, Heather, Jason) for start/mid/end sprint conversations
• sprint issues in GitHub
• product backlog in a google spreadsheet

latest Sloan grant revision

We’ve submitted a revision to our Sloan Foundation grant in response to comments and feedback from them, and to reflect some updated ideas we’ve had.

The biggest change is the budget. I’m close to full-time already because TI is my dissertation. But we’ve boosted Heather’s grant salary to the point where she’d only be 50% supported by her current postdoc, with the other half by the TI grant.

12-month goals

As part of our Sloan Foundation grant process, we were asked to come up with some measurable outcomes. This ended up being a really valuable exercise, and I anticipate we’ll be checking back with these pretty regularly.

We expect not only to reach these goals by April 2013, but also that our chosen metrics will be increasing across the board. Here they are:

• overall visibility: (50k visits, 30k unique visitors, 500 tweets, 30 blog posts, 60 github watchers, 20 forks)
• scholars: embedding or linking to TI reports on their homepage/CV (n=100), some of whom present these in annual reviews or T&P packages (n=25)
• publishers, repositories, and tools: embedding the total-impact widget on articles/datasets (15 organisations)
• researchers: gathering data for research studies using TI (5 in-progress or published papers)

Of course, in keeping with our open and agile approach, we’ll likely end up modifying these some in response to experience and feedback from the community (if you’ve got ideas on how to improve these, we’d love to hear ‘em). But we reckon they’re a pretty good start.

What are metrics good for?

We talk a lot about metrics. And when you do that, there’s always the risk what you’re measuring or why will become unclear. So this is worth repeating, as was reminded in a nice conversation with Anurag Acharya of Google Scholar (thanks Anurag!).

Metrics are no good by themselves. They are, however, quite useful when they inform filters. In fact, by definition filtering requires measurement or assessment of some sort. If we find new relevant things to measure, we can make new filters along those dimensions. That’s what we’re excited about, not measuring for it’s own sake.

These filters can mediate search for literature. They can also filter other things, like job applicants or or grant applications. But they’re all based on some kind of measurement. And expanding our set of relevant features (and perhaps a machine-learning context is more useful here than the mechanical filter metaphor) is likely to improve the validity and responsiveness of all sorts of scholarly assessment.

The big question, of course, is whether altmetrics like tweets, mendeley, and so on are actually relevant features. We can’t now prove that one way or another, although we’re working on it. I do know that they’re relevant sometimes, and I have the suspicion that they will become more relevant as more scholars move their professional networks online (another assumption, but i think a safe one).

And of course, measuring and filtering are only half the game. You also have to aggregate, to the pull the conversation together. Back when citation was the only visible edge in the network, we used ISI et al. to do this. Of course the underlying network was always richer than that, but the citation graph was the best trace we had. But now the the underlying processes—conversations, reads, saves, etc—are becoming visible as well, and there’s  even more value in pulling together these latent, disconnected conversations. But that’s another post :)

thanks, don’t mind if we do.

With most big companies you could hand over your source code and your business plan and they still would not be a threat to you.

-Paul Graham

total-impact at SPARC OA

I (Jason) presented TI at the SPARC Open Access Meeting in Kansas City last week. It was an interesting event, with a mix of in-the-trenches librarians, publishers, institutional repository folk, and business people representing the growing range of  products and services springing up around Open Science. I found the engagement and growth of this latter group encouraging, since it’s where we see TI ending up.

There was ample skepticism about TI: bit publisher reps were very interested, but non-committal, giving the sense that they want to see a more stable legal/org entity before they take the SaaS plunge. Many librarians had initial “it’s a toy, my faculty care only for IF” reactions, although these tended to thaw after more explanation. Both reactions underscore for me the importance of 1) establishing a trustworthy legal identity for TI and 2) continuing to do outreach and research around the idea of altmetrics in general.

There was a lot of encouraging enthusiasm, as well. The TI poster was mobbed. Several repositories expressed heavy interest in embedding TI stats, and some libraries were interested in contributing plugins. Was great to hear folks say “someone isfinallydoing this…it’s just what we’ve been wanting!”

Another highlight was a great chat with John Wilbanks; the more I hear him talk, the more his open-sci insight and knowledge impresses me. Turns out he’s been keeping well abreast of TI and altmetrics, and has good things to say about total-impact’s future prospects, which was great to hear. I also got a chance to talk with smart folks from the Kauffman Foundation at a dinner they set up; they had thoughtful things to say about the Value Of Entrepreneurship, and why a for-profit startup could be the best structure for even a mission-driven project like us.

So, overall an educational and useful event. Thanks to SPARC for putting it on!