Trying to evaluate topic modeling results. Anyone have some rigorous suggestions? Looking at MALLET diagnostic results: article.gmane.org/gmane.comp.ai.…
@scott_bot @mwidner There are quantitative approaches (look at D. Mimno's online vita for leads), but actually ... what Scott said is right.- @scott_bot @Ted_Underwood Besides: too much subjectivity in such work can lead to simply confirming intuitions/biases.
@Ted_Underwood @scott_bot @mwidner This is useful for me. I'm finding that interpreting the results is way harder than the actual modeling.- @Ted_Underwood @scott_bot @mwidner (Although it's entirely possible that modeling would be harder for me if I knew what I was doing.)
- @scott_bot @mwidner You can sometimes discern "artefacts" -- e.g. a topic shaped by arbitrary overlap of names (George Eliot / TS Eliot).
- @miriamkp @Ted_Underwood @scott_bot Right! The modeling is easy, but it's a black box. To understand the box is to interpret better (I hope)
- @mwidner @scott_bot I'd say that topic modeling works if & when you discover something interesting and *new*. Confirming what we know (+)
- @miriamkp Have you looked at @scott_bot 's guided tour? http://www.scottbot.net/HIAL/?p=19113
- @mwidner @scott_bot I have indeed! So useful. I'm getting to the point where I think I might need to read that Blei article.
- @mwidner @scott_bot "works" in an algorithmic, but not in a humanistic sense. (-)
@Ted_Underwood @mwidner What Ted said. Use it for discovery, comparison, navigation, etc., just try to avoid justification.- I'm finding Excel's data charts incredibly helpful. If you tell MALLET to output *everything*, there's lots of data. #topicmodeling
@scott_bot @mwidner @miriamkp @Ted_Underwood yet what we may call "intuition" often conceals iterations of hypothesis & testing- @nmhouston @mwidner @miriamkp @Ted_Underwood True, and concealed selective iteration can lead to selective conclusions (confirmation bias).
@scott_bot @Ted_Underwood @mwidner See the "Reading Tea Leaves" paper by @boydgraber et al.—it discusses a few approaches to evaluation.- Network graph of the topics & authors (including @EileenAJoy, @jeffreyjcohen, and @KarlSteel) in 1 of @punctum_books: dropbox.com/s/v18smyvukx4r…
- @jeffreyjcohen Prevalent topics in the book (labeled by most common words) & relationships among topics and authors' sections.
mwidner
Mike Widner
Academic Technology Specialist at Stanford. ABD in English at UT Austin. Medieval literature, cognitive science, embodiment, genre theory, digital humanities.
Share & notify
Notify
140
