The bumpy road to academia’s side entrance

I have indicated already earlier on this blog that I miss academia, and that I wouldn’t mind moving back into an academic job. I have made some attempts recently and want to reflect on the process here. Spoiler alert: I’ll start a job at the University of Amsterdam very soon!

In my journey back into academic life I have also applied less successfully twice, and for reflection on that, it is probably useful to understand my boundary conditions:

  • I left my academic career in astrophysics now roughly 8 years ago and have not done any pure science since (at least not visibly).
  • I have few, too few papers from three years of postdoc. I left my postdoc position with no intention to go back, and therefore have just dropped all three first-author papers that were in the making on the spot. They were never and will never be published.
  • I am strongly geographically bound. I can commute, but I can not move. Hence, I am bound to local options.

I have spent these last 8 years on data science and gained a fair amount of experience in that field. All that experience is in applied work. I have not done any fundamental research on data science methodology, As an aside, I have of course learned a lot about software development and team work in companies of different sizes. I have seen the process of going from a Proof-of-Concept study to building actual products in a scalable, maintainable production environment (often in the cloud) up close, very close. Much of that experience could be very useful for academia. If I (and/or my collaborators) back then had worked with standards even remotely resembling what is common in industry, science would progress faster, it would suffer much less from reproducibility issues and it would be much easier to build and use science products for a large community of collaborators.

But I digress…. The first application for an assistant professorship connected closely to some of the work I have done in my first data science job. I spent 5,5 years at a healthcare insurance provider, where some projects were about the healthcare side of things, as opposed to the insurance business. The position was shared between a university hospital and the computer science institute. I applied and got shortlisted, to my surprise. After the first interview, I was still in the race, with only one other candidate left. I was asked to prepare a proposal for research on “Data Science in Population Health” and discussed the proposal with a panel. It needed to be interesting for both the hospital as well as for the computer scientists, so that was an interesting combination of people to please. It was a lot of fun to do, actually, and I was proud of what I presented. The committee said they were impressed and the choice was difficult, but the other candidate was chosen. The main reason was supposedly my lack of a recent scientific track record.

What to think of that? The lack of track record is very apparent. It is also, I think, understandable. I have a full time job next to my private/family life, so there is very little time to build a scientific track record. I have gained very relevant experience in industry, which in fact could help academic research groups as well, but you can’t expect people to build experience in a non-academic job and build a scientific track record on the side, in my humble opinion. I was offered to compete for a prestigious postdoc-like fellowship at the hospital for which I could fine-tune my proposal. I respectfully declined, as that was guaranteed to be short-term, after which I would be without a position again. In fact, I was proud to end with the silver medal here, but also slightly frustrated about the main reason for not getting gold. If this is a general pattern, things would look a little hopeless.

As part of my job, and as a freelancer, I have spent a lot of time and effort on educational projects. I developed training material and gave trainings, workshops and masterclasses on a large variety of data science-related topics, to a large variety of audiences. Some of those were soft skill trainings, some were hard skill. Most were of the executive education type, but some were more ‘academic’ as well. When at the astronomical institute at biking distance a job opening with the title “Teaching assistant professor” appeared I was more than interested. It seemed to be aimed at Early Career Scientists, with a very heavy focus on education and education management. Contrary to far most of the job openings I have seen at astronomical institutes, I did not have to write a research statement, nor did they ask for any scientific accomplishment (at least not literally in the ad, perhaps this was assumed to go without saying). They asked for a teaching portfolio, which I could fill with an amount of teaching that must have been at least on par with successful candidates (I would guess the equivalent of 6 ECTS per year, for 3 years on end, and some smaller, but in topic more relevant stuff before that) and with excellent evaluations all across. Whatever was left of the two pages was open for a vision on teaching, which I gladly filled up as well. Another ingredient that would increase my chances was that this role was for Dutch speaking applicants and that knowledge of the Dutch educational system was considered a plus. Score and score. That should have significantly narrowed the pool of competitors. In my letter, I highlighted some of the other relevant experience I gained, that I would gladly bring into the institute’s research groups.

Right about at the promised date (I was plenty impressed!), the email from the selection committee came in! “I am sorry that we have to inform you that your application was not shortlisted.” Without any explanation given, I am left to guess what was the main issue with my application here. I wouldn’t have been overly surprised if I wasn’t offered the job, but I had good hopes of at least a shortlist, giving me the opportunity to explain in person why I was so motivated, and in my view qualified. So, were they in fact looking for a currently practicing astronomer? Was research more important than the job ad made it seem? Is my teaching experience too far from relevant, or actually not (good) enough? Dare I even question whether even this job ad was actually aiming for top-tier researchers rather than for people with just a heart (and perhaps even talent) for teaching? It’s hard to guess what the main reason was, and I shouldn’t try. One thing I am reluctantly concluding from this application is that a job in professional astronomy is hard to get for somebody who has long left the field. I think this vacancy asked for experience and skills that match my profile very well, so not even being shortlisted says a lot to me. Perhaps that’s not grounded, but that’s how it goes with sentiment, I guess. Perhaps a dedicated data science job in astronomy is still feasible, who knows.

In September, I’ll join the University of Amsterdam.

But alas, as said, I have also applied successfully. Yay! The University of Amsterdam (UvA) had an opening for a lead data scientist in the department of policy and strategy. Working for, rather than in higher education was something that previously didn’t really occur to me, but this really sounds like an opportunity to do what I like to do and do well, in the field where my heart is. The UvA is putting emphasis on data literacy in education as well as (inter-disciplinary) research. Big part of the job will be to build and maintain a network inside and outside of the university with data science communities. The Amsterdam Data Science Center fosters research that uses data science methods and meets around the corner. I will strive to take a central, or at least very visible role in that Center and be very close to academic interdisciplinary research! I’m excited! In due time, I’ll report on my experience.

I regret quitting astrophysics

In 2013 I decided to quit my career in astrophysics, move back “home” and become a data scientist. The blog post I wrote about my decision was probably my best read publication as a professional astronomer and it was moving to read all the reactions from people who were struggling with similar decisions. I meant every word in that blog post and I still agree with most of what I said. Now, 7 years after the fact, it is time to confess: I deeply regret quitting.

This post is meant to give my point of view. Many people who left academia are very happy that they did. Here I present some arguments why one might not want to leave, which I hope will be of help for people facing decisions like these.

I miss being motivated. In the first few years after jumping ship many people asked me why I would ever wanted to not be a professional astronomer. I have always said that my day-to-day work wasn’t too different, except that what I did with data was about financial services or some other business I was in, rather than about galaxies and the Universe, but that the “core activities” of work were quite similar. That is kind of true. On an hour by hour basis, often I’m just writing (Python) code to figure things out or build a useful software product. The motivation to do what you do, though, is very very different. The duty cycle and technical depth of projects are short and shallow and the emphasis of projects is much more on getting working products than on understanding. I am doing quite well (in my own humble opinion), but it is hard to get satisfaction out of my current job.

I miss academic research. The seeds of astronomy were planted at very young age (8, if I remember correctly). The fascination for the wonders of the cosmos has changed somewhat in nature while growing up but hasn’t faded. Being at the forefront of figuring things out about the workings of the Universe is amazing, and unparalleled in any business setting. Having the freedom to pick up new techniques that may be useful for your research is something that happened to me only sporadically after the academic years. The freedom to learn and explore are valuable for creative and investigative minds and it doesn’t fit as well in most business settings that I have seen.

I miss working at academic institutions. The vibe of being at a large research institute, surrounded by people who are intrinsically motivated to do what they do was of great value to me. Having visitors over from around the globe with interesting, perhaps related work was a big motivator. That journal clubs, coffee discussions, lunch talks, colloquiums etc. are all “part of the job” is something that even most scientists don’t always seem to fully appreciate. Teaching, at the depth of university level classes, as a part of the job is greatly rewarding (I do teach nowadays!).

I miss passion and being proud of what I do. The internet says I have ”the sexiest job of the 21st century”, but I think my previous job was more enjoyable to brag about at birthday parties. I can do astro as a hobby, but that simply doesn’t give you enough time to do something substantial enough.

I don’t miss … Indeed, the academic career also had its downsides. There is strong competition and people typically experience quite some pressure to achieve. The culture wasn’t always very healthy and diversity and equality are in bad shape in academia. Success criteria of your projects and of you as a person are typically better motivated in business. The obligatory nomadic lifestyle that you are bound to have as an early career scientist were a very enjoyable and educational experience, but it can easily become a burden on your personal life. The drawbacks and benefits of any career path will balance out differently for everybody. If you get to such a point, don’t take the decision lightly.

The people who questioned my decision to become an extronomer were right. I was wrong. It seems too late to get back in. I think I have gained skills and experience that can be very valuable to the astronomical community, but I know that that is simply not what candidates for academic positions are selected on. On top of that, being geographically bound doesn’t help. At least I will try to stay close to the field and who knows what might once cross my path.

Hacking for a future data flood

Astronomy has always been a “big data science”. Astronomy is an observational science: we just have to wait, watch, see and interpret what happens somewhere on the sky. We can’t control it, we can’t plan it, we can just observe in any kind of radiation imaginable and hope that we understand enough of the physics that governs the celestial objects to make sense of it. In recent years, more and more tools that are so very common in the world of data science have also penetrated the field of astrophysics. Where observational astronomy has largely been a hypothesis driven field, data driven “serendipitous” discoveries have become more commonplace in the last decade, and in fact entire surveys and instruments are now designed to be mostly effective through statistics, rather than through technology (even though it is still stat of the art!).

In order to illustrate how astronomy is leading the revolutions in data streams, this infographic was used by the organizers of a hackathon I went to nearing the end of April:
Streams and volumes of data!

The Square Kilometer Array will be a gigantic radio telescope that is going to result in a humongous 160 TB/s rate of data coming out of antennas. This needs to be managed and analysed on the fly somehow. At ASTRON a hackathon was organized to bring together a few dozen people from academia and industry to work on projects that can prepare astronomers for the immense data rates they will face in just a few years.

As usual, and for the better, smaller working groups split up and started working on different projects. Very different projects, in fact. Here, I will focus on the one I have worked on, but by searching for the right hash tag on twitter, I’m sure you can find info on many more of them!

ZFOURGE

We jumped on two large public data sets on galaxies and AGN (Active Galactic Nuclei: galaxies with a supermassive black hole in the center that is actively growing). One of them was a very large data set with millions of galaxies, but not very many properties of every galaxy (from SDSS), the other, out of which the coolest result (in my own, not very humble opinion) was distilled was from the ZFOURGE survey. In that data set, there are “only” just under 400k galaxies, but there were very many properties known, such as brightnesses through 39 filters, derived properties such as the total mass in stars in them, the rate at which stars were formed, as well as an indicator whether or not the galaxies have an active nucleus, as determined from their properties in X-rays, radio, or infrared.

I decided to try something simple and take the full photometric set of columns, so the brightness of the objects in many many wavelengths as well as a measure of their distance to us into account and do some unsupervised machine learning on that data set. The data set had 45 dimensions, so an obvious first choice was to do some dimensionality reduction on it. I played with PCA and my favorite bit of magic: t-SNE. A dimensionality reduction algorithm like that is supposed to reveal if any substructure in the data is present. In short, it tends to conserve local structure and screw up global structure just enough to give a rather clear representation of any clumping in the original high dimensional data set, in two dimensions (or more, if you want, but two is easiest to visualize). I made this plot without putting in any knowledge about which galaxies are AGN, but colored the AGNs and made them a bit bigger, just to see where they would end up:
t-SNE representation of galaxy data from ZFOURGE

To me, it was absolutely astonishing to see how that simple first try came up with something that seems too good to be true. The AGN cluster within clumps that were identified without any knowledge of the galaxies having an active nucleus or not. Many galaxies in there are not classified as AGN. Is that because they were simply not observed at the right wavelengths? Or are they observed but would their flux be just below detectable levels? Are the few AGN far away from the rest possible mis-classifications? Enough questions to follow up!

On the fly, we needed to solve some pretty nasty problems in order to get to this point, and that’s exactly what makes these projects so much fun to do. In the data set, there were a lot of null values, no observed flux in some filters. This could either mean that the observatory that was supposed to measure that flux didn’t point in the direction of the objects (yet), or that there was no detected flux above the noise. Working with cells that have no number at all or only upper limits on the brightness in some of the features that were fed to the machine learning algorithm is something most ML models are not very good at. We made some simple approximations and informed guesses about what numbers to impute into the data set. Did that have any influence on the results? Likely! Hard to test though… For me, this has sprung a new investigation on how to deal with ML on data with upper or lower limits on some of the features. I might report on that some time in the future!

The hackathon was a huge success. It is a lot of fun to gather with people with a lot of different backgrounds to just sit together for two days and in fact get to useful results, and interesting questions for follow-up. Many of the projects had either some semi-finished product, or leads into interesting further investigation that wouldn’t fit in two days. All the data is available online and all code is uploaded to github. Open science for the win!

Leaving the field: becoming an extronomer

I wrote the post below roughly 5 years ago now. I quit my academic job in astrophysics, turned to industry and became a data scientist. I might at some point write something about how I think about all of this now.

I have decided to quit astronomy and start a job in the ‘real world’

I give up on a dream. I thoroughly enjoy doing astronomy. I have time left on my current contract and am even fairly confident that after this and another limited number of temporary jobs, I could have gotten a more permanent job in the field at some time, somewhere. So why quit? The uncertainty in a career in astronomy is enormous. If you don’t belong to the very top, and I don’t, you will have to go with the flow and move to wherever the field wants you. Sometimes that will get you to a very nice place in every respect (as we had in Baltimore), sometimes the place to work is very nice, but the place to live less so (like our current situation) and without a doubt even less desirable combinations are possible. Not the biggest deal for a short term postdoc (though too bad if it really doesn’t work out), but where will this long sought-after tenure-track job take you? And when?

Everybody who has done it for a while knows it: living far away from family and good friends is not easy. You can and will build up a new social life (if you even care about a social life, which you should), but those close at heart will be far. Too far, often. Our daughter deserves to grow up among the love of her grandparents and the rest of her family, just as well as they deserve to witness Amy growing up. It is a choice that everyone has to make for him-/herself, but for me a career in astronomy does not outweigh this aspect of life.

Contrary to (too) many academics, I believe that jobs outside of academia can be equally interesting. In many jobs, the everyday activities are even very similar to those of an astronomer. I think I have landed such a job. I will work in data science and business intelligence at a relatively small scale health insurance provider. An example of a project could be to detect fraud in their databases of claims, doctors, hospitals etc. (automatically). In my opinion, that is intellectually as challenging as the questions I am working on in astronomy, with the additional benefit that more people than just a handful of colleagues care about what you do.

Doing astronomy is fun. Like me, many colleagues often describe it as getting paid for your hobby. I can now go back to doing it purely as a hobby. There are several projects I will try to stay involved in to some extent, and I have a couple of very small projects in mind that I still can do. One doesn’t need to be a professional astronomer to do something fun and remotely useful. As an extronomer, you can easily be an amateur astrophysicist as well.

There are many aspects of working in astronomy that I will miss. The friendly atmosphere, collegiality and informality are a bless. I have met many great friends, with whom I hope to stay in contact. On the other hand, I am very much looking forward to my new profession and old environment. Even though I will leave the field professionally, at heart and in my way of thinking I will always be an astronomer.