On the tractability and approximability of non-submodular cardinality-based $s$-$t$ cut problems in hypergraphs

A minimum $s$-$t$ cut in a hypergraph is a bipartition of vertices that separates two nodes $s$ and $t$ while minimizing a hypergraph cut function. The cardinality-based hypergraph cut function assigns a cut penalty to each hyperedge based on the number of nodes in the hyperedge that are on each side of the split. Previous work has shown that when hyperedge cut penalties are submodular, this problem can be reduced to a graph $s$-$t$ cut problem and hence solved in polynomial time. NP-hardness results are also known for a certain class of non-submodular penalties, though the complexity remained open in many parameter regimes. In this paper we highlight and leverage a connection to Valued Constraint Satisfaction Problems to show that the problem is NP-hard for all non-submodular hyperedge cut penalty, except for one trivial case where a 0-cost solution is always possible. We then turn our attention to approximation strategies and approximation hardness results in the non-submodular case. We design a strategy for projecting non-submodular penalties to the submodular region, which we prove gives the optimal approximation among all such projection strategies. We also show that alternative approaches are unlikely to provide improved guarantees, by showing it is UGC-hard to obtain a better approximation in the simplest setting where all hyperedges have exactly 4 nodes.