CAMML Publication Style

Continuing the very useful discussion we’ve been having on the new conference on Computational and Mathematical Modeling in Linguistics (here and here), I’d like to invite further discussion of the choice to do short paper (6-8pp) submissions instead of the usual abstract submissions for linguistics conferences. We had a little bit of discussion of some pros and cons of this choice on the original post, mostly relating to the potential conflict/competition of publishing something there as opposed to ACL or CogSci. Kyle Rawlins recently raised a number of potential issues with us by email, and I’d like to relay some of these concerns and invite more general discussion of these and other considerations.

To summarize Kyle’s concerns (Kyle, please feel free to comment below to expand on or correct anything):

  • The time and infrastructure burden for reviewers and organizers is substantially more than for reviewing abstracts. This might make it harder to find reviewers and organizers.
  • There is no culture of submitting conference papers for review in linguistics, and it’s a much greater time commitment/risk to prepare a paper to submit to a conference than to prepare an abstract. Could this discourage linguists from submitting? (especially if there is another relevant conference whose submission only requires preparing an abstract?)
  • It’s not clear how such publications should/would count for tenure and hiring purposes in linguistics departments. In many departments only journal publications count, and this kind of publication could preclude a journal publication.

It’s clear that this would be a novel approach for linguistics and that this approach could potentially discourage participation of linguists, which is not our goal. So, the other side of the equation is – is it worth it? What are the advantages of this approach and would these advantages outweigh these or other potential costs? I advocated for paper submission, hoping that peer-review would improve the quality of the work presented at the conference and have the potential to elevate the status of the papers published there as well as the status of the conference itself. Could this status be elevated enough for these papers to count as short journal papers, on par with brief articles or squibs in journals, for purposes of tenure-review and hiring in linguistics departments? And if not, how problematic is this?

What do you think about the relative risks and potential benefits of this approach? What other considerations are there?

6 thoughts on “CAMML Publication Style

  1. Thomas Graf

    I can’t say that I agree with Kyle’s points.

    – Speaking from personal experience, reviewing an 8 page paper takes no more time than reviewing a 2 page abstract because in contrast to the abstract you do not need to read it multiple times. Abstracts are incredibly dense, which makes it very hard to give them a fair and well-reasoned review.

    – Abstract reviews are frustrating to write and to read. The reviewer doesn’t get enough detail to act as more than a gatekeeper that shields the conference from work that appears poorly thought out. That’s not exactly fulfilling work, and it makes the time spent on abstract reviews feel like a slog compared to paper reviews. Due to the lack of detail in abstracts, the reviews often lack the substance to significantly improve the submitted work, which makes them often pointless to the authors (and may also be disheartening to the reviewers).

    – Even if paper reviews took more time, if a smallish conference like MOL manages to review 16 page papers on time, why would linguists struggle with this?

    – I also don’t understand the remark about increased organizational burdens. The paper gets uploaded as a pdf to Easychair (or the ACL platform), just like any abstracts at any other linguistics conference. The reviewers leave their comments in plain text on the submission platform, together with a point rating — again, just like they do for conferences. After a verdict is reached by the organizers, the evaluations are sent to the authors via Easychair — just like abstract reviews.

    – I don’t understand the time commitment/risk part either. If the paper gets rejected, you do exactly what you do with a rejected abstract: revise and submit to another conference. But with a paper you have the advantage that it is much easier to revise based on reviews than an abstract. And if the paper keeps being rejected, share it online to spread the idea. Nobody reads an abstract, people do read manuscripts.

    – The tenure point is also a weak one, because computational linguists in linguistics departments are already in that situation, and it is not killing us. It’s not too hard to explain the format to colleagues who aren’t familiar with it. If anything, the conference could have a positive impact by educating more linguists about alternative publication types.

    As for advantages, there are several (more in-depth and competent reviews, less of a chore for reviewers, higher-quality program, proceedings already published at the time of the conference, distinguishing yourself from other linguistics conferences). But I don’t think we even need to venture into that discussion: abstracts simply will not do.

    Tons of computational work cannot be adequately evaluated based on 2 page abstracts. That’s the very reason none of the formal conferences (ACL, Sigmorphon, tag+, FG, MOL, LACL, CMCL, to name but a few) accept abstracts. Definitions and notation alone often take up more than 2 pages, and there is no fixed standard model like Minimalism or OT that every reviewer can be expected to be familiar with.

    If I may be flippant: if I want to role the dice and see if a watered-down 2 page sketch of my work makes it through the reviewing stage, there’s already tons of linguistics conferences I can submit to. I don’t think that an abstract-based conference would be taken seriously by many computational linguists. Frankly, I wouldn’t consider submitting, and I would have strong doubts about the quality of the program.

    My insistence on paper review may seem somewhat antagonistic, but I similarly feel that painting a well-established type of conference submission as a cause for worry rather than an opportunity just because it deviates from linguistic convention is equally antagonistic. That might not have been Kyle’s intention, but that’s how it comes across to me (and maybe others). This conference, by virtue of its interdisciplinarity, will necessarily have to make compromises — just picking a name was already fraught with difficulties. But I strongly believe that the submission type is not something you can compromise on if you want to attract computational linguists.

    Reply
    1. Kyle Rawlins

      I’m not going to go through this point by point, but I do want to correct a few misconceptions. First, my comments were offered with the perspective of having some interests and experience on both sides of this line, and weren’t intended to be antagonistic to anyone (it’s still quite hard for me to see how they could be). Second, I don’t particularly think that either type of conference acceptance process is a panacea of how to determine quality (both have substantial problems, IMO), but what I do think is that a peer-reviewed journal publication model is preferable for various reasons to one centered around peer-reviewed conference proceedings. In practice the two seem to be mostly incompatible. (Except for things like TACL, a model which isn’t really in the running as far as I know, and is a whole next level of organizational complication.)

      I also want to gently say that it does not at all match my experience to suggest that that reviewing a 2-3 page abstract takes the same time and energy as an 8-12 page paper in ACL style (or similar).

      Reply
      1. Thomas Graf

        The proceedings of the last MOL will have revised and extended versions published as a special issue of the Journal of Language Modelling. The same would be possible for CAMML, assuming there is enough of an interest in a journal publication to devote the required time.

        Reply
  2. Ewan

    Writing papers that have as much useful content as a 30-page paper with all the fat removed seems like a hard thing to argue against.

    Personally, I’m already in the position of having to have my publications counted and shunted into categories by potentially unknowing committees, and that’s just the way it is when you do this kind of work. It’s going to be like that for us regardless of whether this conference exists or not. The solution I came up with is quantity. Publish a lot, and, if you can, guard against committees that aren’t well adjusted to your particular mix of disciplines by trying to get some of each type. That’s already the way it is, so this doesn’t change anything, and it also does linguistics a service by introducing a new and better kind of paper.

    Reply
  3. Gaja Jarosz Post author

    Two post ago, we had already had a brief discussion about whether we should have at least two types of submissions 1) peer-reviewed papers and 2) abstracts, just for talks (or maybe posters), which possibly are open to previously published work. I wonder to what extent this ‘dual-submission’ strategy will help with the cultural-clash issues we’re discussing here. If someone just wants to present their work as a talk, with no proceedings, they have that option, but if they’re happy with the peer-reviewed paper submissions, they can do that too.

    Reply
    1. Thomas Graf

      That sounds good in theory, but in practice you’ll probably run into incommensurability issues unless abstracts are just for posters (or blitz talks). Is a well-reviewed abstract better than a mediocre paper? Is a paper with middling reviews better than a bad abstract (maybe the idea is good and just couldn’t be conveyed well in two pages)? These judgments are already fairly delicate without having two completely different types of submission to consider.

      Another point worth pondering: for the proceedings you need a critical mass of papers. You can’t publish with, say, the ACL, and only have three accepted papers because everybody else opted for abstracts. Just imagine somebody writes a paper and then the proceedings don’t happen, they probably won’t consider submitting to the conference again.

      Would it make sense, then, to restrict abstracts to posters? Well, my impression is that posters are considered much less attractive than talks. A talk has more prestige, and many people also find it less stressful. Basically, I’ve never heard the sentence “good thing I got a poster, I would’ve canceled if I had to give a talk”, but the opposite is more common, and I’ve definitely said it myself before. Why does that matter? Because if the goal is to bridge some kind of chasm between disciplines, restricting abstracts to posters may be perceived as too little return value for those who do not want to write papers. But if you don’t restrict abstracts to posters, you get the problems I pointed out at the beginning.

      Reply

Leave a Reply to Thomas Graf Cancel reply

Your email address will not be published. Required fields are marked *