关于数据库插入数据不重复的有关问题
关于数据库插入数据不重复的问题
问下各位大神我现在碰到一个问题,我在网页上抓取了新闻插入了数据库。但是我今天抓到的和昨天抓到的可能会有所重复。那我想我该怎么去避免把重复的数据再次插入数据库。
说下思路或者有代码最好了。
------解决方案--------------------
如果楼主采用第一条的话,这里附送 删除sql重复数据的方式:
问下各位大神我现在碰到一个问题,我在网页上抓取了新闻插入了数据库。但是我今天抓到的和昨天抓到的可能会有所重复。那我想我该怎么去避免把重复的数据再次插入数据库。
说下思路或者有代码最好了。
------解决方案--------------------
如果楼主采用第一条的话,这里附送 删除sql重复数据的方式:
用SQL语句,删除掉重复项只保留一条
在几千条记录里,存在着些相同的记录,如何能用SQL语句,删除掉重复的呢
1、查找表中多余的重复记录,重复记录是根据单个字段(peopleId)来判断
select * from people
where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)
2、删除表中多余的重复记录,重复记录是根据单个字段(peopleId)来判断,只留有rowid最小的记录
delete from people
where peopleName in (select peopleName from people group by peopleName having count(peopleName) > 1)
and peopleId not in (select min(peopleId) from people group by peopleName having count(peopleName)>1)
3、查找表中多余的重复记录(多个字段)
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
4、删除表中多余的重复记录(多个字段),只留有rowid最小的记录
delete from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)
5、查找表中多余的重复记录(多个字段),不包含rowid最小的记录
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)
6.消除一个字段的左边的第一位:
update tableName set [Title]=Right([Title],(len([Title])-1)) where Title like '村%'
7.消除一个字段的右边的第一位:
update tableName set [Title]=left([Title],(len([Title])-1)) where Title like '%村'