3bij3 – Developing a framework for researching recommender systems and their effects | Amsterdam University Press Journals Online
2004
Volume 2, Issue 1
  • E-ISSN: 2665-9085

Abstract

Abstract

Today’s online news environment is increasingly characterized by personalized news selections, relying on algorithmic solutions for extracting relevant articles and composing an individual’s news diet. Yet, the impact of such recommendation algorithms on how we consume and perceive news is still understudied. We therefore developed one of the first software solutions to conduct studies on effects of news recommender systems in a realistic setting. The web app of our framework (called 3bij3) displays real-time news articles selected by different mechanisms. 3bij3 can be used to conduct large-scale field experiments, in which participants’ use of the site can be tracked over extended periods of time. Compared to previous work, 3bij3 gives researchers control over the recommendation system under study and creates a realistic environment for the participants. It integrates web scraping, different methods to compare and classify news articles, different recommender systems, a web interface for participants, gamification elements, and a user survey to enrich the behavioural measures obtained.

Loading

Article metrics loading...

/content/journals/10.5117/CCR2020.1.003.LOEC
2020-02-01
2024-04-18
Loading full text...

Full text loading...

/deliver/fulltext/26659085/2/1/03_CCR2020.1.003.LOEC.html?itemId=/content/journals/10.5117/CCR2020.1.003.LOEC&mimeType=html&fmt=ahah

References

  1. Arceneaux, K., & Johnson, M.(2013). Changing minds or changing channels? Partisan news in an age of choice. Chicago, IL: University of Chicago Press. doi:10.7208/chicago/9780226047447.001.0001
    [Google Scholar]
  2. Bawden, D., & Robinson, L.(2009). The dark side of information: Overload, anxiety and other paradoxes and pathologies.Journal of Information Science, 35(2), 180–191. doi:10.1177/0165551508095781
    [Google Scholar]
  3. Beam, M.A.(2014). Automating the news: How personalized news recommender system design choices impact news reception.Communication Research, 41(8), 1019–1041. doi:10.1177/0093650213497979
    [Google Scholar]
  4. Bhaskar, M.(2016). In the age of the algorithm, the human gatekeeper is back.The Guardian. Retrieved from https://www.theguardian.com/technology/2016/sep/30/ age-of-algorithm-human-gatekeeper
    [Google Scholar]
  5. Bodó, B., Helberger, N., Eskens, S., & Möller, J.(2019). Interested in diversity: The role of user attitudes, algorithmic feedback loops, and policy in news personalization.Digital Journalism, 7(2), 206–229. https://doi.org/10.1080/21670811.2018.1521292
    [Google Scholar]
  6. Bogers, T., & Van den Bosch, A.(2007). Comparing and evaluating information retrieval algorithms for news recommendation. In Proceedings of the 2007 ACM Conference on Recommender Systems (pp. 141–144). Minneapolis, MA: ACM Press. doi:10.1145/1297231.1297256
    [Google Scholar]
  7. Bomhardt, C., & Gaul, W.(2005). NewsRec, a personal recommendation system for news websites. In C.Weihs & W.Gaul (Eds.), Classification—the ubiquitous challenge (pp. 394–401). Berlin: Springer. doi:10.1007/3‑540‑28084‑7_45
    [Google Scholar]
  8. Bountouridis, D., Harambam, J., Makhortykh, M., Marrero, M., Tintarev, N., & Hauff, C.(2019). SIREN: A simulation framework for understanding the effects of recommender systems in online news environments. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 150–159). Atlanta, GA: ACM Press. doi:10.1145/3287560.3287583
    [Google Scholar]
  9. boyd, D., & Crawford, K.(2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon.Information Communication and Society, 15(5), 662–679. doi:10.1080/1369118X.2012.678878
    [Google Scholar]
  10. Bozdag, E.(2013). Bias in algorithmic filtering and personalization.Ethics and Information Technology, 15(3), 209–227. doi:10.1007/s10676‑013‑9321‑6
    [Google Scholar]
  11. Bridge, D., & Kelly, J.P.(2006). Ways of computing diverse collaborative recommendations. In Lecture Notes in Computer Science (pp. 41–50). Berlin: Springer. doi:10.1007/11768012_6
    [Google Scholar]
  12. Bruns, A.(2018). Gatewatching and news curation: Journalism, social media, and the public sphere. New York: Peter Lang. doi:10.3726/b13293
    [Google Scholar]
  13. Burscher, B., Vliegenthart, R., & De Vreese, C.H.(2015). Using supervised machine learning to code policy issues.The ANNALS of the American Academy of Political and Social Science, 659(1), 122–131. doi:10.1177/0002716215569441
    [Google Scholar]
  14. De Smedt, T., & Daelemans, W.(2012). Pattern for python.Journal of Machine Learning Research, 13(6), 2063–2067.
    [Google Scholar]
  15. Donsbach, W.(2014). Journalism as the new knowledge profession and consequences for journalism education.Journalism, 15(6), 661–677. doi:10.1177/1464884913491347
    [Google Scholar]
  16. Dylko, I., Dolgov, I., Hoffman, W., Eckhart, N., Molina, M., & Aaziz, O.(2017). The dark side of technology: An experimental investigation of the influence of customizability technology on online political selective exposure.Computers in Human Behavior, 73, 181–190. doi:10.1016/j.chb.2017.03.031
    [Google Scholar]
  17. Ekstrand, M.D., & Willemsen, M.C.(2016). Behaviorism is not enough: Better recommendations through listening to users. In Proceedings of the 10th ACM Conference on Recommender Systems (pp. 221–224). Boston, MA: ACM Press. doi:10.1145/2959100.2959179
    [Google Scholar]
  18. Gaines, B.J., & Kuklinski, J.H.(2011). Experimental estimation of heterogeneous treatment effects related to self-selection.American Journal of Political Science, 55(3), 724–736. doi:10.1111/j.1540‑5907.2011.00518.x
    [Google Scholar]
  19. Garcin, F., & Faltings, B.(2013). PEN recsys. In Proceedings of the 2013 International News Recommender Systems Workshop and Challenge (NRS) (pp. 3–9). New York, NY: ACM Press. doi:10.1145/2516641.2516642
    [Google Scholar]
  20. Garcin, F., Faltings, B., Donatsch, O., Alazzawi, A., Bruttin, C., & Huber, A.(2014). Offline and online evaluation of news recommender systems at swissinfo.ch. In Proceedings of the 8th ACM Conference on Recommender Systems (pp. 169–176). Foster City, CA: ACM Press. doi:10.1145/2645710.2645745
    [Google Scholar]
  21. Geiß, S., Magin, M., Stark, B., & Jürgens, P.(2018). “Common Meeting Ground” in Gefahr? Selektionslogiken politischer Informationsquellen und ihr Einfluss auf die Fragmentierung individueller Themenhorizonte.Medien & Kommunikationswissenschaft, 66(4), 502–525. doi:10.5771/1615‑634X‑2018‑4‑502
    [Google Scholar]
  22. Grinberg, M.(2014). Flask web development. Sebastopol, CA: O’Reilly Media.
    [Google Scholar]
  23. Guess, A., & Coppock, A.(2018). Does counter-attitudinal information cause backlash? Results from three large survey experiments.British Journal of Political Science, online first. doi:10.1017/S0007123418000327
    [Google Scholar]
  24. Gupta, J., & Gadge, J. (2015, Jan). Performance analysis of recommendation system based on collaborative filtering and demographics. In International Conference on Communication, Information Computing Technology (pp. 1–6). doi:10.1109/ICCICT.2015.7045675
    [Google Scholar]
  25. Haim, M., Graefe, A., & Brosius, H.B.(2018). Burst of the filter bubble? Effects of personalization on the diversity of Google News.Digital Journalism, 6(3), 330–343. doi:10.1080/21670811.2017.1338145
    [Google Scholar]
  26. Helberger, N.(2019). On the democratic role of news recommenders.Digital Journalism, online first. doi:10.1080/21670811.2019.1623700
    [Google Scholar]
  27. Helberger, N., Karppinen, K., & D’Acunto, L.(2018). Exposure diversity as a design principle for recommender systems.Information Communication and Society, 21(2), 191–207. doi:10.1080/1369118X.2016.1271900
    [Google Scholar]
  28. Honnibal, M., & Montani, I.(2017). spaCy 2: Natural Language Understanding with Bloom Embeddings, Convolutional Neural Networks and Incremental Parsing.To appear.
    [Google Scholar]
  29. Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., & Gay, G.(2007). Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search.ACM Transactions on Information Systems (TOIS), 25(2), 1–27. doi:10.1145/1229179.1229181
    [Google Scholar]
  30. Jonnalagedda, N., Gauch, S., Labille, K., & Alfarhood, S.(2016). Incorporating popularity in a personalized news recommender system.PeerJ Computer Science, 2. doi:10.7717/peerj‑cs.63
    [Google Scholar]
  31. Karakaya, M.Ö., & Aytekin, T.(2017). Effective methods for increasing aggregate diversity in recommender systems.Knowledge and Information Systems, 56(2), 1–18. doi:10.1007/s10115‑017‑1135‑0
    [Google Scholar]
  32. Karimi, M., Jannach, D., & Jugovac, M.(2018). News recommender systems–Survey and roads ahead.Information Processing & Management, 54(6), 1203–1227. doi:10.1016/j.ipm.2018.04.008
    [Google Scholar]
  33. Kitchin, R.(2014). Big Data, new epistemologies and paradigm shifts.Big Data & Society, 1(1), 1–12. doi:10.1177/2053951714528481
    [Google Scholar]
  34. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., & Newell, C.(2012). Explaining the user experience of recommender systems.User Modeling and User-Adapted Interaction, 22(4-5), 441–504. doi:10.1007/s11257‑011‑9118‑4
    [Google Scholar]
  35. Konstan, J.A., & Riedl, J.(2012). Recommender systems: from algorithms to user experience.User Modeling and User-Adapted Interaction, 22(1/2), 101–123. doi:10.1007/s11257‑011‑9112‑x
    [Google Scholar]
  36. Kotkov, D., Wang, S., & Veijalainen, J.(2016). Knowledge-Based systems: A survey of serendipity in recommender systems.Knowledge-Based Systems, 111, 180–192. doi:10.1016/j.knosys.2016.08.014
    [Google Scholar]
  37. Kroon, A., Trilling, D., Fokkens, A., Loecherbach, F., Moeller, J., van Atteveldt, W., & van der Velden, M.(2019). Deriving semantics from dutch media corpora: The Amsterdam word embedding model. Paper presented at Etmaal van de communicatiewetenschap. Nijmegen, Netherlands.
    [Google Scholar]
  38. Kunert, J., & Thurman, N.(2019). The form of content personalisation at mainstream, transatlantic news outlets: 2010–2016.Journalism Practice, 1–22. doi:10.1080/17512786.2019.1567271
    [Google Scholar]
  39. Lee, H., Kwak, N., & Campbell, S.W.(2015). Hearing the other side revisited: The joint workings of cross-cutting discussion and strong tie homogeneity in facilitating deliberative and participatory democracy.Communication Research, 42(4), 569–596. doi:10.1177/0093650213483824
    [Google Scholar]
  40. Leuener, R.(2017). NZZ Companion: How we successfully developed a personalised news application.Medium. Retrieved from https://medium.com/@rouven.leuener/nzz-companion-how-we-successfully-developed-a-personalised-news-app-d3c382767025
    [Google Scholar]
  41. McQuail, D.(2007). Revisiting diversity as a media policy goal. In W. A.Meier & J.Trappel (Eds.), Power, performance and politics: Media policy in Europe (pp. 41–57). Baden-Baden: Nomos. doi:10.5771/9783845202938‑41
    [Google Scholar]
  42. Messing, S., & Westwood, S.J.(2014). Selective exposure in the age of social media: Endorsements trump partisan source affiliation when selecting news online.Communication Research, 41(8), 1042–1063. doi:10.1177/0093650212466406
    [Google Scholar]
  43. Mikolov, T., Corrado, G., Chen, K., & Dean, J.(2013). Efficient estimation of word representations in vector space.Proceedings of the International Conference on Learning Representations (ICLR), 1–12. doi:10.1162/153244303322533223
    [Google Scholar]
  44. Moeller, J., Trilling, D., Helberger, N., Irion, K., & De Vreese, C.(2016). Shrinking core? Exploring the differential agenda setting power of traditional and personalized news media.Info, 18(6). doi:10.1108/info‑05‑2016‑0020
    [Google Scholar]
  45. Möller, J., Trilling, D., Helberger, N., & van Es, B.(2018). Do not blame it on the algorithm: An empirical assessment of multiple recommender systems and their impact on content diversity.Information, Communication and Society, 1–19. doi:10.1080/1369118X.2018.1444076
    [Google Scholar]
  46. Möller, J., van de Velde, R.N., Merten, L., & Puschmann, C.(2019). Explaining online news engagement based on browsing behaviour: Creatures of habit?Social Science Computer Review, 1–17.doi:10.1177/0894439319828012
    [Google Scholar]
  47. Napoli, P.M.(2015). Social media and the public interest: Governance of news platforms in the realm of individual and algorithmic gatekeepers.Telecommunications Policy, 39(9), 751–760. doi:10.1016/j.telpol.2014.12.003
    [Google Scholar]
  48. Nechushtai, E., & Lewis, S.C.(2019). What kind of news gatekeepers do we want machines to be? Filter bubbles, fragmentation, and the normative dimensions of algorithmic recommendations.Computers in Human Behavior, 90, 298–307. doi:10.1016/j.chb.2018.07.043
    [Google Scholar]
  49. Nguyen, T.T., Hui, P.M., Harper, F.M., Terveen, L., & Konstan, J. A.(2014). Exploring the filter bubble. In Proceedings of the 23rd International Conference on World Wide Web (pp. 677–686). doi:10.1145/2566486.2568012
    [Google Scholar]
  50. Nic, N., Fletcher, R., Kalogeropoulos, A., Levy, D.A., & Nielsen, R. K.(2018). Reuters institute digital news report 2018. Reuters Institute for the Study of Journalism. Retrieved from www.digitalnewsreport.org/survey/2018
    [Google Scholar]
  51. Paliouras, G., Mouzakidis, A., Moustakas, V., & Skourlas, C.(2008). PNS: A personalized news aggregator on the web.Studies in Computational Intelligence, 104, 175–197. doi:10.1007/978‑3‑540‑77471‑6_10
    [Google Scholar]
  52. Pariser, E.(2011). The filter bubble: What the Internet is hiding from you. London: Penguin. doi:10.3139/9783446431164
    [Google Scholar]
  53. Pazzani, M.J., & Billsus, D.(2007). Content-based recommendation systems. In The adaptive web (pp. 325–341). Berlin: Springer. doi:10.1007/978‑3‑540‑72079‑9_10
    [Google Scholar]
  54. Peng, H., Liu, J., & Lin, C.Y.(2016). News citation recommendation with implicit and explicit semantics. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (pp. 388–398). doi:10.18653/v1/P16‑1037
    [Google Scholar]
  55. Peperkamp, J., & Berendt, B.(2018). Diversity checker. In Adjunct publication of the 26th Conference on User Modelling, Adaptation and Personalization (UMAP) (pp. 35–41). New York, NY: ACM Press. doi:10.1145/3213586.3226208
    [Google Scholar]
  56. Ribeiro, M.T., Ziviani, N., Moura, E.S.D., Hata, I., Lacerda, A., & Veloso, A.(2014). Multiobjective pareto-efficient approaches for recommender systems.ACM Transactions on Intelligent Systems and Technology, 5(4), 1–20. doi:10.1145/2629350
    [Google Scholar]
  57. Ricci, F., Rokach, L., & Shapira, B.(2011). Introduction to recommender systems handbook. In Recommender systems handbook (pp. 1–35). Boston, MA: Springer US. doi:10.1007/978‑0‑387‑85820‑3_1
    [Google Scholar]
  58. Schlesinger, P., & Doyle, G.(2015). From organizational crisis to multi-platform salvation? Creative destruction and the recomposition of news media.Journalism: Theory, Practice & Criticism, 16(3), 305–323. doi:10.1177/1464884914530223
    [Google Scholar]
  59. Shah, D.V., Cappella, J.N., & Neuman, W.R.(2015). Big Data, digital media, and Computational Social Science.The ANNALS of the American Academy of Political and Social Science, 659(1), 6–13. doi:10.1177/0002716215572084
    [Google Scholar]
  60. Shani, G., & Gunawardana, A.(2011). Evaluating recommendation systems. In Recommender systems handbook (pp. 257–297). Boston, MA: Springer US. doi:10.1007/978‑0‑387‑85820‑3_8
    [Google Scholar]
  61. Sidorov, G., Gelbukh, A., Gómez-Adorno, H., & Pinto, D.(2014). Soft similarity and soft cosine measure: Similarity of features in vector space model.Computacion y Sistemas, 18(3), 491–504. doi:10.13053/CyS‑18‑3‑2043
    [Google Scholar]
  62. Song, H., Jung, J., & Kim, Y.(2017). Perceived news overload and its cognitive and attitudinal consequences for news usage in South Korea.Journalism & Mass Communication Quarterly, 94(4), 1172–1190. doi:10.1177/1077699016679975
    [Google Scholar]
  63. Stroud, N.J.(2011). Niche news: The politics of news choice. New York, NY: Oxford University Press.
    [Google Scholar]
  64. Sunstein, C.R.(2009). Republic. com 2.0. Princeton, NJ: Princeton University Press.
    [Google Scholar]
  65. Teppan, E.C., & Zanker, M.(2015). Decision biases in recommender systems.Journal of Internet Commerce, 14(2), 255–275. doi:10.1080/15332861.2015.1018703
    [Google Scholar]
  66. Thorson, K., & Wells, C.(2016). Curated flows: A framework for mapping media exposure in the digital age.Communication Theory, 26(3), 309–328. doi:10.1111/comt.12087
    [Google Scholar]
  67. Thurman, N., & Schifferes, S.(2012). The future of personalization at news websites: Lessons from a longitudinal study.Journalism Studies, 13(5-6), 775–790. doi:10.1080/1461670X.2012.664341
    [Google Scholar]
  68. Trilling, D.(2014). Weg vom manuellen Speichern: RSS-Feeds in der automatisierten Datenerhebung bei Onlinemedien. In K.Sommer, M.Wettstein, W.Wirth, & J.Matthes (Eds.), Automatisierung in der Inhaltsanalyse (pp. 73–89). Köln: Herbert von Halem.
    [Google Scholar]
  69. Trilling, D., Van Klingeren, M., & Tsfati, Y.(2017). Selective exposure, political polarization, and possible mediators: Evidence from the Netherlands.International Journal of Public Opinion Research, 29(2), 189–213. doi:10.1093/ijpor/edw003
    [Google Scholar]
  70. Trilling, D., van de Velde, B., Kroon, A.C., Loecherbach, F., Araujo, T., Strycharz, J., Jonkman, J.(2018). INCA: Infrastructure for content analysis. In 14th International Conference on e-Science (IEEE) (pp. 329–330). Amsterdam. doi:10.1109/eScience.2018.00078
    [Google Scholar]
  71. Van Atteveldt, W., Strycharz, J., Trilling, D., & Welbers, K.(2019). Toward open computational communication science: A practical roadmap for reusable data and code.International Journal of Communication, 13(1), 3935–3954.
    [Google Scholar]
  72. Van Atteveldt, W., & Peng, T.Q.(2018). When communication meets computation: Opportunities, challenges, and pitfalls in computational communication science.Communication Methods and Measures, 12(2-3), 1–12. doi:10.1080/19312458.2018.1458084
    [Google Scholar]
  73. Victor, P., De Cock, M., & Cornelis, C.(2011). Trust and recommendations. In Recommender systems handbook (pp. 645–675). Boston, MA: Springer. doi:10.1007/978‑0‑387‑85820‑3_20
    [Google Scholar]
  74. Yang, J. A.(2016). Effects of popularity-based news recommendations (most-viewed) on users exposure to online news.Media Psychology, 19(2), 243–271. doi:10.1080/15213269.2015.1006333
    [Google Scholar]
  75. Zhang, S., Yao, L., Sun, A., & Tay, Y.(2019). Deep learning based recommender system: A survey and new perspectives.ACM Computing Surveys (CSUR), 52(1), 5. doi:10.1145/3285029
    [Google Scholar]
  76. Zuiderveen Borgesius, F.J., Trilling, D., Möller, J., Bodó, B., de Vreese, C.H., & Helberger, N.(2016). Should we worry about filter bubbles?Internet Policy Review, 5(1), 1–16. doi:10.14763/2016.1.401
    [Google Scholar]
  77. Řehůřek, R., & Sojka, P.(2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (pp. 45–50). Valletta, Malta: ELRA.
    [Google Scholar]
http://instance.metastore.ingenta.com/content/journals/10.5117/CCR2020.1.003.LOEC
Loading
/content/journals/10.5117/CCR2020.1.003.LOEC
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): computational social science; news; recommender systems; web application
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error