A team effort

By Emily Margosian, assistant editor, April 1, 2021

Although the term “crowdsourcing” was officially coined by Wired in 2006, the practice has long had deep roots within science. One of the earliest surviving examples is the Audubon Society’s annual Christmas Bird Count, which began in 1900 and continues to this day, and relies on data gathered by citizen scientists to study population bird trends (J Gen Intern Med. 2014; 29: 187-203). 

However, at the present moment in history, the urgency of the ongoing COVID-19 pandemic has shone a spotlight on crowdsourcing’s potential as a powerful tool to gather and share data. “When you are in crisis, you need to have a broad understanding of what’s happening worldwide very quickly. That’s where crowdsourcing can really provide strength,” said Esther Freeman, MD, PhD, associate professor of dermatology at Harvard Medical School, and principal investigator of the COVID-19 Dermatology Registry. “As a single provider caring for a couple of patients, it’s hard to see overall trends. When you’re able to bring together lots of observations quickly, you can start to form hypotheses that inform the pandemic front line. If we can recognize a new sign or symptom that can help people identify COVID and stop transmission, that’s a win.” 

This month, experts speak with DermWorld about how crowdsourcing has gained new momentum during COVID-19, and how it may reshape medical research after the pandemic. 

Crowdsourcing on the rise

By definition, crowdsourcing — an open call for voluntary assistance from a large group of individuals — has been present in medicine for some time. However, as COVID-19 has altered daily life at an unprecedented rate, it became evident early in the pandemic that traditional research and publishing models were unable to accommodate the speed of the unfolding disaster. “At the time, none of this was appearing yet in the formal literature. A lot of information was being transmitted through social media platforms like Twitter or Facebook,” recalled Dr. Freeman. “There was so much happening so quickly; the traditional publishing system was not necessarily the way that we were going to be able to understand what was happening at the front lines across the world in such a rapidly moving pandemic.” 

To help formalize what had been organically taking place across the web, various medical specialties began forming registries to collect case reports of observations related to COVID-19. “At the very beginning of the pandemic, we were trying to sort out many different issues related to COVID and what it meant for dermatology as a specialty,” said Lindy Fox, MD, professor of dermatology at University of California, San Francisco School of Medicine, and member of the COVID-19 Dermatology Registry’s design and concept team. “We were starting to hear from colleagues in Europe about cutaneous manifestations of COVID, so it seemed like a good time to start trying to collect some of that information. My background is in complex medical and inpatient dermatology, so one of the things I like to do is recognize new patterns of cutaneous disease and identify new issues as they emerge.” 

“In many ways, Pandora's box has been opened. I don't think we're ever going to quite go back.”

Launched in March 2020, the COVID-19 Dermatology Registry is among a growing number within the specialty — including registries focused specifically on psoriasis, atopic dermatitis, hidradenitis suppurativa, alopecia, and others. “The dermatology registry community has really grown during this time; we have collaboration now between a number of different groups,” said Dr. Freeman. 

In response to the public rollout of several COVID vaccines, the registry has continued to expand its scope of data collection. “As of right now, we have over 400 vaccine manifestations in the registry, and by the time you are reading this, it will be many more. I think it’s important for us to be nimble as COVID-19 asks different things of us in different phases of the epidemic,” said Dr. Freeman. “Being able to respond as a dermatologic community and think about how we can assist in the house of medicine is important.” 

Building a registry: Nuts and bolts  

Prior to COVID, Dr. Freeman was involved with the development of several other international databases, a process she says typically takes about six months. “The fact that we were able to launch the registry in nine days — from proposal to actually accepting cases — I think is really a testament to the level of international collaboration that has occurred during the pandemic. It’s been a huge change in how we react to medical emergencies just by force of necessity.” 

Devon McMahon, a Harvard Medical School student, was the COVID-19 Dermatology Registry’s primary programmer. “I’ve been Dr. Freeman’s medical student research fellow for approximately the past two years. Prior to COVID-19, we had worked closely on various epidemiological projects on Kaposi sarcoma in Kenya and Uganda, and from those I had gained skills in helping to create databases and conduct statistical analyses,” she said. 

According to McMahon, the blueprint for the COVID-19 Dermatology Registry was created in collaboration with other pre-existing registries developed for rheumatology and inflammatory bowel disease. “We then collaborated with leaders on the Academy’s COVID-19 Task Force to create a series of dermatology-specific questions about COVID-19 dermatologic manifestations and past dermatologic history,” she said. “Once we had designed the questions, actually programming the registry itself was a relatively quick process due to my previous global health work with Dr. Freeman.” “Working with trainees on this project has been particularly fulfilling, because in addition to providing rapid data analysis back to frontline health care providers, we are training the next generation of dermato-epidemiologists,” said Dr. Freeman.

Privacy and data security

In an era of rampant information theft, particularly in health care, members of the COVID-19 Dermatology Registry team said that data security and patient privacy were among their top concerns. “We used a platform called REDCap to create and host the registry, which is secured through our home institution, Massachusetts General Hospital (MGH), and can only be accessed by the research team,” said McMahon. “Additionally, all information in the registry is de-identified, meaning that we do not ask for any identifying patient information such as name, address, medical record number, or date of birth.”

Allowing non-dermatologists to contribute was a strategic decision, according to Dr. Freeman. “At the beginning of the pandemic, the majority of COVID-19 patients were being treated by frontline health care providers, so we felt it was important to reach a broad swath of providers who saw these patients when they came in, as opposed to limiting it just to dermatologists who might be seeing them a couple of weeks later,” she said. 

While registry participation is not limited to dermatologists, only health care workers can contribute. To ensure data quality, each entry undergoes a vetting process, explained McMahon. “For every case entered, our trained research team looks at the complete record to check for quality. We require health workers entering cases to state their professional title and give their institutional email address, so it is often clear when a patient has mistakenly entered a case.”  

Data access

In addition to ensuring information was safely stored and protected, the registry’s team had to make decisions about access to the data being collected. “Data access is an issue that’s near and dear to my heart. I firmly believe that this data belongs to everyone in the dermatology community,” said Dr. Freeman. “We’re so indebted to all of the providers across the globe who have entered data into the registry and trusted us with their cases.” 

Currently vetted access to the data is allowed by following a series of data request protocols. “For anyone who is interested in using the registry data to do a study, you can visit the AAD website and submit a request to view the data with a specific project in mind,” said Dr. Fox, who also serves as a member of the Academy’s Data Governance Workgroup. Learn more.

“You tell us how you’re planning on analyzing the data and exactly which variables you might need to answer your scientific question. Then the Data Governance Workgroup will review the proposal and give you some feedback. Once we’re all in agreement, we share the data with you for analysis,” said Dr. Freeman. “We do ask you to have a statistical analysis plan as part of the data request process, and we require that you have either an epidemiologist or a biostatistician involved in the project. The reason for this is that these are large and complex datasets that have changed over time, and there’s a lot of nuance in terms of how to handle them appropriately.” 

Advantages of crowdsourcing

One undeniable advantage of crowdsourcing is its speed. Particularly in times of crisis, large amounts of data can be tracked in real-time during the beginning stages of an epidemic or outbreak — potentially fast-tracking life-saving science. According to Dr. Fox, several insights drawn from registry data have already begun to inform clinical practice when caring for dermatology COVID patients. “One example we’ve gathered is that pernio-like lesions or ‘COVID toes’ are usually a sign of having had infection or exposure; it’s a sequela, but for most patients it’s an indication that they are not actively infectious. Kind of a, ‘don’t worry, this is a good sign,’ when you’re talking to patients,” she said. “Alternately, we’ve learned that common rashes may be a sign of active COVID infection. If you have a patient coming to you with a pityriasis rosea-like rash, you’ll want to check if COVID is one of the reasons that they have it. In the hospital setting, if you see retiform purpura, that may be a sign that somebody is doing pretty poorly; they’re having abnormal clotting processes that are associated with COVID and may indicate much more severe disease. Essentially, the registry helps us understand the severity of the disease, prognostically how a patient will do, and guides our workup and counseling of patients.” 

Even beyond the end of the pandemic, crowdsourced data may yield helpful insights into the long-term effects of the virus. “I think we will be dealing with the after-effects of COVID for a long time,” said Dr. Freeman. “We are collecting data on COVID ‘long-haulers’ and expect to be learning more about this population over time.” 

Crowdsourcing, by nature, also facilitates collaboration across specialties, institutions, and the globe, potentially reframing what’s possible for science to accomplish. “The fact that we were able to program and go live so quickly is really a testament to the fact that large organizations, like the Academy and the ILDS, were willing to be incredibly nimble and move rapidly in response to something like this,” said Dr. Freeman. “We have a huge group of collaborators, many of whom I’ve never worked with before the pandemic. These folks have really become my ‘COVID silver lining’ — dermatology faculty across the world who all united behind this idea. It’s shown that the global community can come together very rapidly when needed.” 

Limitations of crowdsourcing

Despite its advantages, crowdsourcing as a means of data collection invariably has its limits. The newly coined phrase “big data hubris” has been used to describe overconfidence in the quality of crowdsourced information. One such example is Google’s ill-fated Flu Trends, which attempted to forecast flu prevalence based on web searches of symptoms. Dubbed an “epic failure” by Wired in 2015, the project missed the peak of the 2013 flu season by a whopping 140% due to problems with its tracking algorithm. 

In an effort to maintain data quality within the COVID-19 Dermatology Registry, the decision was made to exclude patient submissions. “We do occasionally have a patient who tries to send their case, and that’s actually relatively easy to identify and remove those cases from the registry,” said Dr. Freeman. “It’s not to say there isn’t a role for registries where patients enter their own data, but the quality is definitely more difficult to deal with, especially when it comes to categorizing the morphology of skin rashes in the beginning of the pandemic. It was a very conscious decision to be provider-facing so that we could go for higher quality data.” 

Dr. Freeman is open about the limitations of crowdsourced data, which she says are not a replacement for traditional research studies. “A registry is never going to be as good or detailed as a very well-designed cohort study with high-level data. As a PhD-trained epidemiologist, I’m very aware of study design, and I think it’s important not to overstep what a registry can do,” she explained. “You have to think of a registry as a giant case series; it has the same issues as any other case series would have. We don’t have a denominator; I can’t tell you from the registry what percentage of people with COVID are going to develop skin findings. People often ask me that, and I say I truly cannot tell you from the registry; we are just a collection of cases. You need a very well-designed, population-based study to answer that question.”

Looking forward: Crowdsourcing after COVID

As the demands of the COVID era have partially disrupted conventional research models, once the virus is under control, will science return to business as usual? “In many ways, Pandora’s box has been opened,” said Dr. Freeman. “I don’t think we’re ever going to quite go back. I suspect in the future it won’t just be for epidemics. As new medications get rolled out, and as we have seasonal disease trends, I think crowdsourcing can be helpful to track that.” 

Dr. Fox agreed. “COVID has basically put on warp speed items that may have happened anyway, albeit much more slowly. For example, how to make a vaccine very fast, how to change hospital policy, how to implement teledermatology, how to suddenly get paid for teledermatology when that was really hard to do before the pandemic — and I think crowdsourcing falls within that as well. I see it being applied in many different ways moving forward, not only during catastrophic events. We’ve been able to gather so much information and put out useful conclusions in a record time; we’re only going to move forward.”  

“Medical publication is in a constant state of change, just as science is. There is constant innovation, and that will continue at an accelerated pace.”

Dirk Elston, MD, editor of the Journal of the American Academy of Dermatology (JAAD), agreed that the effects of crowdsourcing during the pandemic will likely have an impact on medical research going forward. “Medical publication is in a constant state of change, just as science is. There is constant innovation, and that will continue at an accelerated pace. The ultimate crowdsourced data is registry data — it gives the ability to answer questions that cannot be answered otherwise,” he said. “Currently, JAAD offers expedited review and rapid dissemination with early posting of uncorrected proofs of manuscripts shortly after acceptance. This is becoming more common to allow rapid dissemination of knowledge.” 

According to Dr. Fox, while crowdsourcing will never replace peer review as a gold standard, the future may produce an interesting marriage between the two. “We still rely on peer review for critical review of our work. I don’t think that will change. What will be interesting to consider moving forward is if something needs to be put out right away because it’s so critical and may change the way we practice — we may begin to see more things like medRxiv (pronounced MedArchive) where you’re able to get important information out there before it’s peer reviewed. I think that’s going to be very useful in the future.”

Additional DermWorld Resources